DEV Community: payintlab

PSP Change Drift Watch #2: turning the changelog watch into a runnable monitor

payintlab — Tue, 22 Sep 2026 03:31:26 +0000

What this series does: watch Stripe, PayPal and Adyen for API changes, SDK majors
and ecosystem issues, and write up only the ones that can silently break a live
integration. Issue 1 covered five drift findings by hand. Issue 2 turns the
manual watch into a small, runnable tool.

The problem with doing it by hand

Issue 1 listed five changes worth acting on (PayPal IPN migration, Stripe
billed_until, stripe-node 21, Adyen v72, and a few low-priority notes).
Producing that list meant reading changelogs and release notes across three
platforms, then judging which changes could "return 200 while the business logic
never runs".

Doing that by hand has two failure modes:

You only notice drift after it already broke something. That is the whole point of these bugs: they are silent.
You forget to re-check. A changelog read in September is stale by October.

The six pain classes I keep seeing (duplicate events, field-format drift, lost
callbacks, signature failures, new event types, sandbox/production mismatch)
share one root cause: the platform changed and your system did not follow.
So the useful primitive is "detect the change early", not "read the changelog
occasionally".

What I built

A small monitor, psp-drift-watch, that turns the watch into a repeatable job:

sources.json — the watchlist: changelogs, migration guides and SDK release notes, each tagged with a platform and a signal type.
fetch.py — fetches every source and fingerprints the content.
diff.py — compares against the last snapshot and writes a drift report.
a regression hook — optionally runs the payment-qa-framework test suite when something changes.

The fingerprint is a content hash, plus ETag / Last-Modified when the server
provides them, so any edit to a watched page shows up as a drift entry on the
next run.

Honest about what it does and does not do

This is a monitor, not a detector. It tells you "this page changed", not
"this change breaks you". The judgment still lives in the three signals from
Issue 1:

EOL or migration — there is a deadline, but the silent failures live in the migration window.
Field default changes — no error, the value just disappears.
SDK major with a pinned API version — the upgrade itself hides the risk.

A changed source is a prompt to apply that judgment (plus the regression
checklist), not a verdict.

Run it

git clone https://github.com/PayIntLab/psp-drift-watch.git
cd psp-drift-watch

python3 fetch.py                 # first snapshot
cp snapshot.json baseline.json   # freeze the baseline

./drift.sh                       # fetch + diff + report
./drift.sh --regress             # also trigger the regression suite

The report (drift-report.md) lists changed / new / failed / unchanged sources,
plus the pre-upgrade regression checklist.

Regression checklist (unchanged from Issue 1)

Pin the API version; upgrading is an explicit step, not a side effect of a dependency update.
Run success, decline, expiry, refund, dispute and duplicate-callback cases.
Assert in tests whether a missing field raises or silently continues.
Cover subscription changes separately: create, renew, cancel, plan change.
Trigger the same event once in sandbox and once in production, and confirm both behave the same.

Over the years I have done both development and testing for cross-border
payment systems: payment integrations and SDK upgrades, test automation
frameworks, payment flow testing, dropped-order triage, and reconciliation
fallbacks. I now focus on payment verification for products going global, and I
also take on integration and automation work. Happy to compare notes.

Repo

https://github.com/PayIntLab/psp-drift-watch

A silently dropped webhook, recovered five minutes later: a reconciliation drill you can run locally

payintlab — Mon, 14 Sep 2026 04:09:08 +0000

Webhooks are at-least-once, not exactly-once. They get retried, duplicated, delayed, and sometimes they never arrive at all. When that happens the money has moved but the merchant order has not, and nobody notices until a customer complains.

I built a small drill that reproduces the worst version of this on purpose, on a laptop, in about a minute of real time:

The merchant creates a Stripe order in a local emulator.
The emulator drops payment_intent.succeeded for that payment.
The provider intent is succeeded, while the merchant order stays PENDING — money moved, business state did not.
The sandbox clock advances five minutes. The scheduled reconciliation task pulls the payment intent, sees succeeded, and recovers the order: status=PAID, paidSource=reconciliation.

The recording shows the drill steps on the left and the matching provider and merchant log lines on the right. Every step has its own log lines and starts with a dashed separator and a short pause: order created, intent confirmed, dropped delivery, zero webhook deliveries, clock advanced, and finally the reconciliation scan that recovers the order.

The whole project is a multi-provider payment sandbox: Stripe-style, PayPal-style, card acquirer and crypto emulators, plus a reference merchant that verifies signatures and handles webhooks idempotently.

Why reconciliation, and not just retries

Retries only help when the delivery reaches you and your handler fails. They do not help when:

the webhook is dropped or lost,
your service is down for the whole retry window,
the endpoint was misconfigured and every attempt was rejected,
the platform itself has an incident.

In all of those cases the provider still knows the truth. A scheduled pull that compares local state against provider state is the fallback that closes the gap.

What the drill shows

merchant order : order_102
payment intent : pi_1001
provider says  : succeeded
merchant says  : PENDING
webhook        : never arrived (dropped by the emulator)

...five minutes of sandbox time later...

status         : PAID
paid source    : reconciliation
recovered at   : 2026-09-11T08:57:16Z

summary
  webhook deliveries   : 0 (dropped)
  reconciliation scans : 1
  orders recovered     : 1

Run it locally:

git clone https://github.com/pqa-labs/pqa-payments-sandbox
cd pqa-payments-sandbox
./scripts/reconciliation-drill.sh

Design notes worth copying

Reconciliation is pull-based and idempotent. A second scan does not touch an already recovered order. There is a test for the double-recovery case.
Provider status is the source of truth: succeeded (Stripe), COMPLETED (PayPal), CAPTURED (card).
Recovered orders carry paidSource=reconciliation, so you can tell webhook-driven state from reconciliation-driven state in reports and audits.
The sandbox clock makes a five minute window visible in seconds, which is what makes the demo watchable.
One deliberate gap: crypto deposits are skipped because the merchant stores a deposit address, not a queryable deposit id. Finding and fixing that is a good exercise.

What to test in your own system

Drop a webhook on purpose and confirm the order still ends up correct.
Deliver the same webhook twice and confirm nothing is written twice.
Restart the service in the middle of the retry window and check recovery.
Assert that a recovered order is not recovered twice.
Make sure a reconciliation run is visible in metrics or logs, not silent.

The repo is pqa-labs/pqa-payments-sandbox. It is a local sandbox, not a production system, and it should never be used with real payment data.

I spent years doing both development and testing for cross-border payment systems: payment integrations and SDK upgrades, test automation frameworks, payment flow testing, dropped-order triage, and reconciliation fallbacks. I now focus on payment verification for products going global, and I also take on integration and automation work. Happy to compare notes.

PSP Change Drift Watch #1: PayPal IPN migration, Stripe API versions, Adyen v72

payintlab — Tue, 08 Sep 2026 02:27:44 +0000

What this series does: watch Stripe, PayPal and Adyen for API changes, SDK majors and ecosystem issues, and write up only the ones that can silently break a live integration. Issue 1 covers five.

Bottom line
Only one item really needs action now: if you still run PayPal IPN or Website Payments Standard, schedule a migration audit before January 2027. Migration is not the hard part; the hard part is that four layers change at once during the window, and each layer can "return 200" while the business logic never fires. The Stripe and Adyen items below are "run regression before you upgrade", not urgent like the PayPal one.

Figure 1 IPN to Webhooks four breakpoints

Figure 1: moving from IPN to Webhooks changes four layers at once, and each one can silently drop a payment.

PayPal IPN is in its migration window: the easiest way to lose money silently Source: PayPal Orders v1-to-v2 upgrade guide + community analysis

Timeline: PayPal stopped issuing new IPN/WPS credentials at the end of 2025, marked Website Payments Standard deprecated in January 2026, and shuts it all down in January 2027. The replacement is REST Webhooks.

The deadline is not the danger. The migration window is. Switching from IPN to Webhooks is not changing an endpoint URL; four layers change at the same time:

Payload format: form-encoded becomes JSON. An old urlencoded parser receives JSON and every field comes back undefined; the handler still returns 200, PayPal treats it as delivered and stops retrying, and the order stays "unpaid".
Field paths: mc_gross becomes resource.amount.value (still a string, not a number); custom becomes custom_id; payer_email is not even present on the capture resource and needs a second order lookup.
Status words and events: Completed becomes COMPLETED, and a refund is a separate event type (PAYMENT.CAPTURE.REFUNDED). If you only subscribe to COMPLETED, refunds and disputes never arrive.
Signature verification: the _notify-validate round-trip becomes offline RSA-SHA256 verification. Code that kept the old round-trip path fails verification on every webhook; a "pass if no signature" dev branch is worse, because it turns the webhook endpoint into a public interface.
Quick self-check:

grep -rn "payment_status|mc_gross|payer_email|_notify-validate" .
Any hit means IPN-era code is still around. While you change it, also check: are refund/dispute events subscribed, is the amount explicitly converted to a number, and has the idempotency key moved from txn_id to the event id.

Stripe API 2026-05-27.dahlia: billed_until is no longer returned by default Source: Stripe changelog

On SubscriptionItem, billed_until changed from "included by default" to "only present if you expand it". Code that reads this field directly (reconciliation, renewal display, dunning) gets an empty value after the upgrade, often without an error.

Figure 2 beforeafter comparison for billeduntil

Figure 2: the same code read this field before the upgrade and reads null after it, with no error raised.

Self-check before upgrading:

Look at Dashboard > Developers > API version. Do not let the SDK silently move the pinned version.
Search the codebase for billed_until and confirm every use expands it.
Run the subscription cases: create, renew, cancel, plan change, and assert billed_until has a value.

Stripe SDK majors: the wrong webhook parser now throws instead of failing silently Source: stripe-node 21 release notes (pins 2026-03-25.dahlia)

stripe-node 21 has an easy-to-miss breaking change: constructEvent and parseEventNotification now check the payload type, throw when you pass the wrong one, and point you to the other method. Before, using the wrong parser did not raise; it just returned an object with missing data, which is the "signature passes but no data comes out" class of silent failure. After upgrading to 21, the same code starts throwing. Also remember that v2 thin events do not carry full data; you still need to fetch the event with an API key.

Self-check: confirm whether your events are v1 or v2. v1 goes through constructEvent; v2 goes through parseEventNotification plus a data fetch. After the upgrade, trigger the same event in sandbox and in production, and make that a required test rather than only running the happy path.

Adyen Checkout v72: stricter validation and clearer error codes Source: Adyen upgrade notes (April 2026)

v72 is not a silent breaker; it turns vague failures into explicit errors, which is a good direction, but it is still a behavior change for older integrations. Malformed requests that used to be tolerated now fail outright. The default Pix session validity was also extended to 24 hours, which interacts with timeout logic.

Self-check: before upgrading, run the failure paths once each (decline, expiry, retry, partial payment) and confirm the new error codes line up with your retry branches. Do not test only the success path.

Low priority, worth a note PayPal order receiptJson structure changed in March 2026; merchants using the Shopify PayPal channel should check where they parse it. PayPal Enterprise Payouts now exposes direct-debit return reasons through webhooks and a GET endpoint; reconciliation folks can wire it up. Stripe 2026-07-29.preview surfaces some requirements errors earlier. Do not run preview versions on production. How to decide whether a change is worth covering Watch for three signals:

EOL or migration: there is a deadline, but the real risk lives in the migration window, where silent failures are most likely (PayPal IPN here).
Field default changes: no error, the value is just suddenly missing (billed_until here).
SDK major with a pinned API version: the upgrade itself hides the risk (Stripe SDK here).

Figure 3 the three signals

Figure 3: a change that can "return 200 while the business logic never runs" is worth an issue entry with a self-check.

Watch these sources: official platform changelogs, SDK release notes, issues in open-source integration libraries, and payment developer communities. If a change could "return 200" without the business logic running, it belongs in the next issue.

General regression checklist before any upgrade
Pin the API version; upgrading is an explicit step, not a side effect of a dependency update
Run success, decline, expiry, refund, dispute and duplicate-callback cases
Assert in tests whether a missing field raises or silently continues
Cover subscription changes separately: create, renew, cancel, plan change
Trigger the same event once in sandbox and once in production and confirm both behave the same
Over the years I have done both development and testing for cross-border payment systems: payment integrations and SDK upgrades, test automation frameworks, payment flow testing, dropped-order triage, and reconciliation fallbacks. I now focus on payment verification for products going global, and I also take on integration and automation work. Happy to compare notes.

Sources (Issue 1)
PayPal Orders v1-to-v2 migration guide: https://developer.paypal.com/api/rest/integration/orders-api/v1-v2-migration
PayPal IPN deprecation analysis: https://dev.to/flarecanary/paypal-ipn-is-deprecated-the-ipn-webhooks-migration-drops-payments-in-four-silent-ways-1afp
Stripe billed_until change: https://docs.stripe.com/changelog/dahlia/2026-05-27/subscription-item-billed-until-includable
stripe-node 21 release notes (webhook parser throws): https://newreleases.io/project/npm/stripe/release/21.0.0
stripe-node PR #2616 (wrong parser throws): https://github.com/stripe/stripe-node/pull/2616
Adyen Checkout v72: https://www.adyen.com/the-latest/upgrade-to-checkout-v72-for-more-reliable-payment-flows