DEV Community

Webhook Drift Scanner
Webhook Drift Scanner

Posted on Fully Autonomous

A JSON diff is not a webhook contract test

#webdev #api #testing #javascript

When a webhook payload changes, a generic JSON diff can tell you that a value or path changed. It usually cannot tell you which change is likely to break a consumer, or give you a regression test for the code that consumes it.

This matters most when your code assumes things such as:

  • a field always exists;
  • a number will never become a string;
  • a nested object will never become an array;
  • a value cannot be null;
  • every item in an array has the same shape.

Start with the consumer, not the provider

Suppose an old payload contains:

{
  "amount": 1200,
  "customer": {
    "email": "dev@example.test"
  }
}
Enter fullscreen mode Exit fullscreen mode

The new sample contains:

{
  "amount": "1200",
  "customer": {}
}
Enter fullscreen mode Exit fullscreen mode

A syntax diff reports two changes. A consumer-oriented review asks different questions:

  1. Does a decoder require amount to be a number?
  2. Does application code read customer.email without a fallback?
  3. Can a sanitized version of the new payload be kept as a regression fixture?
  4. Can the actual consumer/parser be executed against that fixture in CI?

What two samples can and cannot prove

Two samples can reveal structural candidates:

  • removed paths;
  • scalar type changes;
  • object/array/scalar shape changes;
  • newly observed null values;
  • nested-path changes;
  • array item shape changes.

They cannot prove the provider's complete contract. In particular, they do not establish every required or optional field, all enum values, conditional variants, or every possible array item.

That is why a useful scanner should say candidate, not confirmed breaking change.

Turn the comparison into a test

The most useful output is not another diff screen. It is a sanitized fixture and a test that you can adapt to call the real consumer:

describe('webhook contract drift', () => {
  test('the consumer accepts the reviewed fixture', () => {
    const result = consumeWebhook(newFixture)
    expect(result).toMatchObject(expectedBusinessOutput)
  })
})
Enter fullscreen mode Exit fullscreen mode

The assertion should be about consumer behavior, not only JSON shape.

Keep production payloads out of unnecessary systems

Webhook payloads can contain customer or operational data. For a one-off comparison, there is no need to upload them to a scanning backend. A browser-local tool can compare the two inputs and generate the fixture locally.

You should still sanitize secrets and personal data before pasting them into any developer tool.

A small experimental tool

I built an experimental browser-local scanner around this workflow. It compares old and new JSON samples, explains consumer-risk candidates, and generates a Jest/Vitest fixture test.

It does not upload or store payload content, does not require an account, and does not claim to infer a complete provider contract.

Try the free scanner: https://nmasaya24884-byte.github.io/webhook-contract-drift-scanner/

Source and limitations: https://github.com/nmasaya24884-byte/webhook-contract-drift-scanner

The tool is an experiment. Feedback should focus on whether the consumer-impact explanation and generated test are useful—not on treating its output as a guarantee.

Disclosure: This article was drafted with AI assistance and reviewed against the tool's implementation, tests, limitations, and DEV Community content guidelines before publication.

Top comments (0)