Contract-Testing My Own Zerodha Clone: What Specmatic Found in Code I Thought Was Done

Introduction

I have a MERN-stack Zerodha clone in my portfolio — a real-time trading dashboard with a Node/Express backend, a React frontend and dashboard, MongoDB, and Razorpay for payments. It worked. It demoed well. Like most side projects, "working" mostly meant "worked the last time I clicked through signup → login → place an order → pay."

During my internship at Specmatic, most of my days go into working with contract testing — turning OpenAPI specs into executable tests that verify an API actually honors what it promises. Starting the contract testing journey with my own project — the obvious experiment became: what happens if I point the same approach at something I'd already called 'finished'?

This post is that experiment — the setup, the real bugs it surfaced, and the CI gates it left behind. Repo: priya3054/zerodha-specmatic.

Architecture

The application:

• Backend: Node.js / Express, port 3002
• Frontend + Dashboard: React
• Database: MongoDB
• Payments: Razorpay

The contract: a single OpenAPI 3.0 spec at contracts/openapi.yaml, with examples — some inline, most external — under contracts/openapi_examples/.

How Specmatic fits in: specmatic.yaml (Config v3) points at the spec, the running backend, and the examples directory, then generates and runs test scenarios from all three:

version: 3

systemUnderTest:
  service:
    definitions:
      - definition:
          source:
            filesystem:
              directory: contracts
          specs:
            - spec:
                id: zerodhaApiSpec
                path: openapi.yaml
    runOptions:
      openapi:
        type: test
        baseUrl: "http://localhost:8080"
        # reports routes actually registered on the real backend, so Specmatic
        # can flag "Missing In Spec" / "Not Implemented" during coverage analysis
        actuatorUrl: http://backend:3002/actuator/mappings
        workflow:
          ids:
            "POST /create-order -> 200":
              extract: "BODY.id"
    data:
      examples:
        - directories:
            - ./contracts/openapi_examples

specmatic:
  settings:
    test:
      schemaResiliencyTests: all
  governance:
    successCriteria:
      enforce: true
      maxMissedOperationsInSpec: 0
      minCoveragePercentage: 100

Two lines carry most of the weight here. schemaResiliencyTests: all tells Specmatic to stop testing only the requests I thought to write examples for, and instead generate every schema-valid and schema-invalid variation itself — wrong types, boundary values, missing fields — and assert the API rejects what it should. minCoveragePercentage: 100 with maxMissedOperationsInSpec: 0 turns the contract into a gate: an endpoint the spec documents but the backend never implements (or vice versa) fails the build, not just a silent gap in a report.

Everything runs through docker-compose.yml on the open-source specmatic/specmatic:2.49.1 image — Docker is the only local dependency.

From that one spec and config, Specmatic currently generates and runs 178 tests across 20 operations, at 100% API coverage — the vast majority of them schema-resiliency variations I never wrote by hand. The HTML report breaks it down per path: /auth/signup alone gets 2 positive tests for its 200 and 23 generated negative tests asserting the 400 path, /auth/login gets 8 tests hammering the 401 path with malformed credentials, and so on.

What Specmatic actually found

This is the part I didn't expect. I assumed running contract tests on my own, already-working project would mostly be a formality. Instead, the first real runs surfaced bugs I'd never have found by clicking through the UI:

/newOrder had no input validation at all. The original handler saved req.body straight to MongoDB — no type checks, nothing. The contract, meanwhile, already declared qty as an integer with a minimum of 1:

# contracts/openapi.yaml
qty:
  type: integer
  minimum: 1
  example: 5

Schema resiliency mutated qty to 5.5, 0, "5", and worse, and the handler accepted every one of them. The fix that's in the backend now checks type, integer-ness, and range explicitly before anything touches the database:

// backend/index.js
app.post("/newOrder", async (req, res) => {
  const { name, qty, price, mode } = req.body;

  if (
    typeof name !== "string" ||
    typeof qty !== "number" ||
    !Number.isInteger(qty) ||
    qty < 1 ||
    typeof price !== "number" ||
    !["BUY", "SELL"].includes(mode)
  ) {
    return res.status(400).json({ error: "Invalid order payload" });
  }
  // ...proceeds to save the order
});

And the paired example that pins this down as an actual test case, not just a code comment — Specmatic matches request/response examples by name and shape, so a qty: 0 request has to sit next to its expected 400:

// contracts/openapi_examples/placeNewOrder_application_json_400_application_json_1.json
{
  "http-request": {
    "method": "POST",
    "path": "/newOrder",
    "body": { "name": "INFY", "qty": 0, "price": 1555.45, "mode": "BUY" }
  },
  "http-response": {
    "status": 400,
    "body": { "error": "Invalid order payload" }
  }
}

/auth/login could crash on malformed input. Non-string username/password values reached passport.authenticate completely unguarded. Fixed with an explicit type check before authentication ever runs:

// backend/routes/auth.js
router.post("/login", (req, res, next) => {
  if (typeof req.body.username !== "string" || typeof req.body.password !== "string") {
    return res.status(401).json({ error: "Password or username is incorrect" });
  }
  passport.authenticate("local", (err, user) => { /* ... */ })(req, res, next);
});

/verify-payment accepted invalid amount values. A null or string amount crashed the DB update with a 500; a boolean silently coerced to 0/1 and the payment still came back "success." The handler now validates amount is a real number before it ever reaches User.findByIdAndUpdate:

// backend/index.js
if (typeof amount !== "number" || Number.isNaN(amount)) {
  return res.status(400).json({ status: "failure" });
}

Signup wasn't idempotent. Re-running the same contract test twice failed on duplicate-user creation — a problem that only shows up once you're running the same suite repeatedly, which manual testing never does.

None of these came from QA. They came from schemaResiliencyTests: all doing exactly what it's supposed to do: generating the malformed, unexpected, "nobody would type this by hand" requests that a real client eventually will.

Catching a breaking change before it merges

The other half of the setup is the backward-compatibility gate, and I wanted to see it actually catch something rather than take it on faith. I changed qty in the contract from integer to string on a branch called breaking-qty-change — the kind of change that looks harmless in a diff and breaks every existing consumer.

The check that catches it runs in CI (Gate 1), diffing the current spec against a committed baseline copy in baseline/contracts/openapi.yaml:

# .github/workflows/01-contract-repo-ci.yml (Backward compatibility check)
- name: Backward compatibility check (Specmatic)
  run: |
    docker run --rm \
      -v ${{ github.workspace }}:/usr/src/app \
      --entrypoint sh \
      specmatic/specmatic:2.49.1 \
      -c '
        mkdir -p /tmp/compat-check/contracts && cd /tmp/compat-check
        git init -q && git checkout -q -b main
        git config user.email [e-mail] && git config user.name CI
        cp /usr/src/app/baseline/contracts/openapi.yaml contracts/openapi.yaml
        git add contracts/openapi.yaml && git commit -q -m baseline
        git checkout -q -b current
        cp /usr/src/app/contracts/openapi.yaml contracts/openapi.yaml
        git add contracts/openapi.yaml && git commit -q -m current
        specmatic backward-compatibility-check --base-branch=main --target-path=contracts/openapi.yaml
      '

With the breaking change in place, the check fails immediately and points at the exact line:

REQUEST.BODY.qty (contracts/openapi.yaml:208:17)
"type string in the new specification, but type number in the old specification"

And once reverted, the same check passes cleanly across every operation:

Verdict for spec /tmp/compat-check/contracts/openapi.yaml:
  (COMPATIBLE) The spec is backward compatible with the corresponding spec from main

Files checked: 1 (Passed: 1, Failed: 0)

That's 40 compatibility checks across 19 operations on every push — and the breaking qty change never merged. This is exactly the scenario the gate exists for: not "does my backend match my spec right now," but "am I about to ship a change that breaks whoever's already calling this API."

Mocking Razorpay instead of hitting it

Payments are the one dependency I didn't want anywhere near a test run. Instead of a hand-rolled mock, I used Specmatic's service virtualization to stub Razorpay directly from the contract — a razorpay-stub service that comes up alongside mongo and the real backend in docker-compose.yml. Contract tests exercise the full path — auth, order creation, payment verification — without real Razorpay credentials anywhere in CI.

The same mechanism, pointed at contracts/openapi.yaml instead, gives a stub profile that serves a fake backend on port 9000 — useful independent of testing, since it means dashboard work can happen against realistic responses before the real endpoint exists.

Proving multi-step flows, not just single endpoints

A single-endpoint contract test can't tell you whether an ID returned by one call actually gets used correctly by the next one. specmatic.yaml's workflow.ids block handles exactly that for the funds flow — extracting the order ID from POST /create-order's response so it can be threaded into POST /verify-payment:

workflow:
  ids:
    "POST /create-order -> 200":
      extract: "BODY.id"

This runs as its own CI step (Gate 2), separate from the coverage-gated contract run:

# .github/workflows/02-provider-ci.yml
- name: Run workflow test (create-order -> verify-payment ID propagation)
  run: |
    docker compose --profile workflow up \
      --build --abort-on-container-exit --exit-code-from specmatic-workflow-test

For the full user journey — signup, login, place an order, pay — I modeled it as an Arazzo workflow (ZerodhaTradeFlow.arazzo.yaml), built visually in Specmatic Studio and exported: signup → check username → login → get-me → stream live events → get holdings → get positions → place order → create order → verify payment, chained in one run. Each step declares a success condition ($statusCode == 200) and its outputs feed the next step's inputs — the Razorpay order ID from createOrder flows into verifyPayment the same way a real user session would carry it.

Running it end to end: 11 steps, 11 passing, 100% workflow coverage.

Studio also turned out to be the fastest way to manage the example files themselves — it lists every path/response pair in the contract, shows which named example backs it, and can generate and validate new examples against the schema without hand-writing JSON:

Wiring it into CI

Three gates run on every push, using the direct docker run/docker compose form so a red CI run reproduces exactly with the same command locally:

1. Contract Repo CI — Spectral lint, example validation, and the backward-compatibility check above.
2. Provider CI — the contract test plus schema resiliency tests against the real backend, gated on 100% coverage via the actuatorUrl route-reporting endpoint, followed by the workflow test.
3. Consumer CI — consumer stub tests against the frontend.

The actuatorUrl piece is worth calling out on its own: Specmatic can only flag "the backend implements a route the contract doesn't document" if it has a way to ask the backend what it actually implements. I added a small /actuator/mappings endpoint that reports every registered Express route in the same shape Spring Boot's actuator uses, purely so the coverage gate has something to compare the spec against:

// backend/index.js
app.get(["/actuator", "/actuator/mappings"], (req, res) => {
  const dispatcherServlet = listRegisteredRoutes().map(({ path, methods }) => ({
    handler: path,
    predicate: `{${methods.join(",")} [${path}]}`,
    // ...
  }));
  res.json({ contexts: { application: { mappings: { dispatcherServlets: { dispatcherServlet } } } } });
});

Without it, maxMissedOperationsInSpec: 0 has nothing to check against — coverage gates are only as good as the signal you feed them.

Challenges Faced

1. Workflow ID extraction only reaches body fields, not path parameters (Specmatic v1.14.2). The Zerodha API passes everything through request bodies, so the funds flow works end to end — but it's a version constraint worth knowing: check what your pinned version supports before designing around a feature the latest docs describe.

2. Authenticating contract tests without weakening them. Endpoints like /verify-payment check req.isAuthenticated(), but hundreds of generated tests can't each do a login round trip — and disabling auth under test would hide real bugs. The fix: a NODE_ENV === 'test' middleware that accepts a fixture header for an allowlist of known test users only. Random usernames Specmatic invents still hit the real 401 path, so the negative tests stay honest.

3. Express has no actuator, and the coverage gate needs one. maxMissedOperationsInSpec: 0 can only flag undocumented routes if the backend reports what it implements — free in Spring Boot, absent in Express. I built a /actuator/mappings shim that walks Express's router stack and mimics Spring's format. Without it, "100% coverage" would only mean "100% of what the spec happens to mention."

4. Stateful endpoints break when the suite runs twice. The second full run failed on signup — the user from the first run already existed. 178 tests mutate state, and the suite has to survive its own side effects. Making signup idempotent under test is what made repeated CI runs reliable.

5. Infinite streams don't fit finite HTTP assertions. /events is Server-Sent Events — a connection that never closes, which a contract test can't assert against. Under test, the endpoint emits one sample event and closes, keeping it documented and coverage-counted instead of carved out as an untestable exception.

Benefits Achieved

Real bugs found in code I considered done. Five validation and crash bugs that months of manual testing never touched — every one found by a generated negative test, not a scenario I thought to write.

A permanent safety net against breaking changes. The backward-compatibility gate has already blocked one breaking change (qty number → string) from merging, with a failure message pointing at the exact file and line. Future consumers are protected on every push, with zero ongoing effort.

A CI pipeline with no secrets in it. Razorpay is stubbed from its own contract, so the complete payment path runs on every push without a single real credential in GitHub Actions.

The spec became a guarantee, not documentation. With 100% coverage enforced and the actuator flagging undocumented routes, contracts/openapi.yaml provably matches the running backend on every commit — anyone can build against it without reading the backend source.

Frontend development decoupled from the backend. The contract-driven stub serves realistic responses on port 9000, so dashboard work needs no MongoDB, no seed data, no running backend — switching to the real service is a one-URL change.

178 tests I didn't write and don't maintain. The suite regenerates from the spec on every run. New endpoint in the contract means new tests automatically — no test files to keep in sync, no suite that rots as the API evolves.

Key Learnings

1. The happy path is a tiny fraction of your API's real surface. Of the 178 tests, maybe a dozen are scenarios I could have written by hand. The rest — 23 malformed-signup variations, 8 bad-credential login probes — are requests nobody writes examples for, and exactly where all five bugs lived.

2. "Accepts bad input" is worse than "crashes on bad input." The login crash was the loud bug; the scary one was /verify-payment silently coercing a boolean amount into a "successful" payment. Crashes get noticed; silent coercion corrupts data quietly — and negative testing is the only systematic way to find the quiet kind.

3. A contract that isn't executable drifts into fiction. My spec already declared qty as type: integer, minimum: 1 — the backend just never enforced it. The only spec that stays true is one where code-spec disagreement fails a build.

4. Testability is an application feature, not a test concern. Idempotent signup, the SSE test mode, the actuator endpoint — none live in test files. They're changes to the application, made so it could be verified. Contract testing made the code more disciplined, not just more tested.

5. In a payments flow, boundary validation is the whole game. Every fixed bug reduces to trusting req.body. The fixes were trivial — typeof, Number.isInteger, an enum check. The expensive part was finding out which checks were missing, and that's exactly what schema resiliency automated.

6. Make the tool prove itself before you trust it. I deliberately broke the contract on a branch to watch the compatibility gate fail, then reverted to watch it pass. A gate you've only ever seen pass might be silently checking nothing.

Conclusion

I went into this expecting a formality — point the internship tooling at a portfolio project, collect a green checkmark. What I got was proof that "it works when I use it correctly" and "it's safe to expose to a client I don't control" are two very different claims — and manual testing only ever establishes the first one.

The final accounting: five real bugs fixed, one breaking change blocked before merge, and 178/178 tests passing at 100% coverage across 20 operations, an 11-step Arazzo workflow, and three green CI gates on every push — with the entire suite generated from a single OpenAPI file rather than written by hand.

If you're working on any project with an API surface, point this at code you already trust. The setup is a spec file, a config file, and Docker. Clone the repo, run docker compose --profile ci up, and see what your own "finished" project has been hiding:

Repo: github.com/priya3054/zerodha-specmatic