Integrating Specmatic Contract Testing into Spring PetClinic REST: What a "Simple" DELETE Test Taught Me About Test Architecture

Introduction

Hello, I am Kuruba Ramesh. As part of my ongoing work with the Specmatic Full Stack AI Engineering Internship, I integrated contract testing into spring-petclinic-rest — the official Spring team's REST version of the classic Petclinic sample, built with an API-first approach and a complete OpenAPI 3.0 spec.

This post covers four things: the architecture of the setup, exactly what I did step by step, the real gaps I found in Specmatic itself along the way (not just bugs in the application), and how I ultimately designed the CI pipeline around the one gap that couldn't be closed outright.

Repository: https://github.com/KRameshr/spring-petclinic-rest

Architecture

The application: A Spring Boot REST API (Java 17, Spring Data JPA, H2 in-memory database) managing a veterinary clinic — owners, pets, vets, visits, and specialties. 18 API paths, 36 operations, full CRUD.

The contract: The project already ships with an OpenAPI 3.0 spec (src/main/resources/openapi.yml) — a machine-readable document describing every endpoint, every request/response shape, and every possible status code. For example, here's what it declares for fetching a single owner:

/owners/{ownerId}:
  get:
    operationId: getOwner
    parameters:
      - name: ownerId
        in: path
        schema:
          type: integer
          minimum: 1
          maximum: 10
    responses:
      200:
        description: Owner details found and returned.
      404:
        description: Owner not found.

This single block is enough for Specmatic to generate multiple test scenarios: a valid request expecting 200, an out-of-range ID expecting 404, and so on — all without anyone writing a line of test code.

How Specmatic fits in: Specmatic reads the OpenAPI spec and generates test scenarios from it automatically — no test code written by hand. It sends real HTTP requests to the application (already running, on localhost:9966) and checks whether every response matches what the spec promises. This is contract testing: verifying the app honours its own documented contract, not just that it "works."

The configuration (specmatic.yaml, Config V3) ties it all together:

version: 3
specmatic:
  settings:
    test:
      schemaResiliencyTests: all
      maxTestRequestCombinations: 1
systemUnderTest:
  service:
    $ref: "#/components/services/petclinicService"
components:
  services:
    petclinicService:
      definitions:
        - definition:
            specs:
              - spec:
                  path: openapi.yml
      data:
        examples:
          - directories:
              - src/main/resources/openapi_examples
  runOptions:
    petclinicServiceTest:
      openapi:
        baseUrl: http://localhost:9966/petclinic/api

This points Specmatic at the spec file, the base URL of the running app, a folder of external examples for controlling test data, and enables schema resiliency testing — generating negative/edge-case variations, not just the happy path.

The wiring: a JUnit 5 test class (ContractTest.java) implements Specmatic's SpecmaticContractTest interface. Running mvn test -Dtest=ContractTest reads this configuration, loads the spec, generates scenarios, and runs them.

Step by Step: What I Did

1. Verified the existing setup actually worked.
The project already had specmatic.yaml (Config V3), a JUnit 5 test class, and external examples. First step was simply running it and getting a real, reproducible baseline: 221 scenarios, some passing, some not.

2. Investigated every failure — not just made the build green.
POST /visits failed for every generated request. Tracing it: the request schema was missing a required field (petId) that the database enforced. This was a real spec-vs-database mismatch, not a testing artifact.

3. Fixed the missing field.
Added a request-only schema (VisitCreate), a MapStruct mapping method, and a controller update:

// VisitMapper.java
@Mapping(target = "id", ignore = true)
@Mapping(source = "petId", target = "pet.id")
Visit toVisit(VisitCreateDto visitCreateDto);

Re-ran the suite — this failure was gone for good.

4. Confirmed schema resiliency testing was active.
This is what surfaced most of what follows — negative/edge-case variations catch mismatches the happy path alone never would.

5. Added a dictionary for realistic test data.
Instead of random garbage strings for names and addresses, I wrote openapi_dictionary.yaml with real-looking values (names, cities, phone numbers).

6. Verified the dictionary — and found it wasn't actually being applied.
Specmatic's log confirmed it loaded the file:

Using dictionary file /usr/src/app/src/main/resources/openapi_dictionary.yaml

But checking an actual generated request showed the values were still random:

{
    "firstName": "v-Sc-YI",
    "lastName": "M.",
    "city": "XNPBY"
}

The dictionary keys (Owner, Vet) didn't match the real schema names (OwnerFields, VetFields) — a silent mismatch with no warning. Fixing the key names fixed it:

# before
Owner:
  firstName: [James, Mary, Robert, Patricia]

# after
OwnerFields:
  firstName: [James, Mary, Robert, Patricia]

Re-checked a generated request — real values now:

{
    "firstName": "Patricia",
    "lastName": "Johnson",
    "city": "Madison"
}

This unexpectedly exposed a second bug: POST /visits was returning 201 while the spec said 200. Fixed that too.

7. Found and fixed a caching feature the spec promised but the code never built.
The spec declared an ETag header for conditional GET requests. The app never sent one. Added one file:

@Configuration
public class ETagConfig {
    @Bean
    public Filter shallowEtagHeaderFilter() {
        return new ShallowEtagHeaderFilter();
    }
}

Verified real 304 Not Modified responses now work, with zero controller changes needed across any of the 18 endpoints.

8. Traced the remaining 6 DELETE failures to their root cause — five separate attempts.
Schema ID bounds, a named example, dedicated seed rows, coverage-focused 404 examples, and a dictionary override for the ID. Every attempt hit the same wall (detailed in the Gaps section below): SpecmaticContractTest runs all generated scenarios as a single dynamic JUnit 5 stream with no per-scenario reset hook, so once any scenario mutates real seed data — a DELETE that actually succeeds, for instance — every later scenario touching that row is exposed to the change:

API: DELETE /owners/(ownerId:number) -> 200
  >> RESPONSE.STATUS
      R0002: HTTP status mismatch
      Specification expected status 200 but response contained status 404

9. Closed the gap by designing around the constraint instead of fighting it.
Once the root cause was confirmed five separate ways, I stopped looking for a trick that would avoid it and instead rebuilt around it:

• Made the seed data fully dense within its valid ID bounds — no gaps. A gap anywhere in a bounded range is exactly what schema-driven boundary and negative-mutation tests probe first, so it's a guaranteed eventual collision, not just a risk.
• Split the resiliency-testing configuration into its own CI job. schemaResiliencyTests: all generates randomized boundary and negative-mutation tests across every operation, including DELETE, and is fundamentally incompatible with fixed, dedicated test records when there's no fixture isolation between scenarios — there's always some chance a random DELETE mutation targets the same row as a named example before that example gets its turn. So the pipeline now runs two jobs: contract-tests (named examples only, deterministic, required) and resilience-tests (full resiliency, continue-on-error: true, reports honestly without blocking the pipeline).

Result: contract-tests now passes 100%, reliably, confirmed across many repeated runs both locally and in CI.

10. Found a genuine Hibernate/JPA bug by refusing to accept "flaky" as an explanation.
While bringing the rest of the test suite (all unit and integration tests across all four ClinicService variants — H2/JDBC, HSQL/JDBC, JPA, Spring Data JPA) to a fully passing state, two of those variants had one test failing intermittently: deleting a PetType sometimes threw

TransientPropertyValueException: Persistent instance of 'Pet' references
an unsaved transient instance of 'PetType'

Tracing it instead of writing it off as test-order flakiness found this in the actual repository code:

public void delete(PetType petType) {
    this.em.remove(this.em.contains(petType) ? petType : this.em.merge(petType));
    // ... then, moments later:
    this.em.createQuery("DELETE FROM PetType WHERE id=" + petTypeId).executeUpdate();
}

An entity-level em.remove() and a bulk JPQL DELETE on the same row, in the same method. Bulk JPQL statements force an intermediate flush of the persistence context, and if any other Pet still held an in-memory reference to that PetType, Hibernate correctly refused to proceed rather than silently corrupting state. The fix: remove the redundant em.remove() call — the bulk delete already does the job. The exact same bug existed in both the JPA and Spring Data JPA repository implementations, likely copy-pasted at some point in the project's history.

11. Documented everything, including what didn't work and why.
The README covers every fix, the failed attempts, and the final CI design — with actual log excerpts, not just summaries.

Specmatic Gaps I Found

This is the part I think is most useful to share — not application bugs, but real limitations in Specmatic itself (open-source edition) that I ran into.

Gap 1: No hook to reset state between dynamically generated scenarios.
SpecmaticContractTest / SpecmaticJUnitSupport generates all scenarios as a single dynamic JUnit 5 test stream. Standard JUnit lifecycle hooks like @BeforeEach only apply to individual @Test methods and cannot attach to this stream. configureTest() runs once, for the entire suite — not per scenario. I confirmed this five separate ways — schema bounds, a named example, dedicated seed rows, coverage-focused 404 examples, and a dictionary override — all reproducing the identical cascade once a real ID could be targeted. Full fixture isolation between scenarios is an Enterprise-only feature; in the open-source tier, the correct response is to design your data and CI gates so the constraint can't cause a false failure — dense, gap-free ID ranges plus a separate, non-blocking job for resiliency testing — rather than keep searching for a way around it.

Gap 2: Dictionary key mismatches fail silently.
If a dictionary's top-level key doesn't exactly match the OpenAPI schema name, Specmatic doesn't apply those values — and doesn't warn you either. It simply falls back to random generation, while the log still reports the dictionary file as "loaded." This cost real debugging time, since "loaded successfully" reads like confirmation that it's working.

Gap 3: Dictionary values don't apply to path parameters.
I tried forcing a specific ID via the dictionary to make a DELETE scenario target a known-safe row. Confirmed this doesn't work — dictionary overrides only apply to schema body fields, not path parameters.

Gap 4: Automatic dictionary generation is Enterprise-only.
Specmatic can generate a dictionary automatically from a spec and existing examples — but only in the Enterprise edition. The open-source edition requires writing dictionaries by hand.

Gap 5: The --filter CLI option isn't available through the JUnit5/Maven integration.
Specmatic's CLI supports excluding specific operations (e.g. --filter="!(METHOD='DELETE')") from a run, which would have been a clean way to skip DELETE from resiliency testing specifically. That option isn't exposed through SpecmaticJUnitSupport. I tried passing it as a system property (-DFILTER=...); it had no effect on the JUnit run, and in one test actually correlated with more failures rather than acting as a no-op — worth knowing before assuming it will transfer over from CLI usage.

Gap 6: Writing a reliable 304/ETag example is hard when the underlying value is dynamic.
Even after fixing the app to genuinely support ETags, writing a Specmatic example for the 304 case is awkward — the ETag is a hash of the response body, so it changes whenever seed data is mutated elsewhere in the same test run. A hardcoded example expires the moment other tests touch the same data.

Key Learnings

"The log says it loaded" is not the same as "it's being used." Always verify against an actual generated request, not just the tool's own status message.

Fixing one bug can unmask another. The POST /visits status mismatch was invisible until the dictionary fix let requests actually reach the server.

Repeat an experiment before trusting the conclusion. Five independent attempts at the same DELETE problem, all failing identically, is what turned a guess into a documented, confident root cause.

"Intermittent" is a description of the symptom, not a diagnosis. Both the dictionary key mismatch and the later Hibernate flush-order bug looked like tooling flakiness right up until they were traced to a specific, deterministic line of code.

Some limitations are real and should be designed around, not endlessly worked around. Once a constraint is confirmed — not assumed — the better use of time is building around it honestly (a continue-on-error resiliency job alongside a fully deterministic required gate) rather than continuing to search for a trick that avoids it.

A bug duplicated across two files is a strong signal of copy-paste history. Finding the identical flush-order mistake in both JpaPetTypeRepositoryImpl and SpringDataPetTypeRepositoryImpl was a good reminder to check sibling implementations whenever a bug turns up in one.

Current Status

• Specmatic ContractTest: 254 scenarios, 100% deterministic pass on the required contract-tests CI gate; a separate resilience-tests job runs full schema resiliency testing and reports honestly, non-blocking.
• Full mvn verify: 491/491 tests passing — every REST controller test, all four ClinicService variants, plus config and validator tests — with JaCoCo coverage checks passing.
• API contract coverage: 60%, up from 42% at the start of this work.
• All three GitHub Actions workflows (Specmatic Contract Tests, Java CI, Docker Hub build) are green.

Full technical detail — including exact log excerpts for every experiment above, the complete list of bugs fixed, and the CI design decisions — is in the repository README.

Conclusion

The most valuable part of this project wasn't fixing the obvious bugs — it was the discipline of verifying tool behaviour instead of trusting it, confirming a root cause multiple times before accepting it, and then, once a real limitation was confirmed, designing around it honestly instead of continuing to chase a workaround. That discipline is what turned six unexplained DELETE failures into a fully deterministic, 100%-passing required test gate, and what turned an "intermittent" test into a genuine, fixed Hibernate bug duplicated across two files.

Repository: https://github.com/KRameshr/spring-petclinic-rest