I turned on my integration tests after 50 days and found three bugs 202 green tests couldn't see

Day 50 is the last day of OrderHub — a Spring Boot order-fulfilment system I built one feature at a time, from a single REST controller to eight services with Kafka, sagas, an outbox, Redis, Prometheus and a semantic search index.

For the finale I wanted one command that walks a single order through the whole thing. Getting there meant paying off a debt I'd been carrying since Day 9, and that turned out to be the most useful hour of the entire series.

The debt

Since Day 9 the repo had integration tests. Testcontainers, a real PostgreSQL, a real Redis, the full HTTP stack. Sixteen of them across four files. OrderApiIT. OrderRepositoryIT. IdempotencyIT. RateLimitIT.

Not one of them had ever run.

Maven's surefire plugin — the one that runs your tests — matches three filename patterns: *Test.java, Test*.java, *Tests.java. Notice what's missing. A file called OrderApiIT.java matches none of them. It compiled on every build, sat in the source tree looking like coverage, and was never executed.

That's the failure mode worth internalising. Not a red build. Not a skipped test with a warning. Silence. The build was green, the tests existed, and the two facts had nothing to do with each other.

The IT suffix isn't arbitrary — it's the convention for failsafe, surefire's sibling for slow tests. Two differences matter:

1. It matches the other naming convention (IT*.java, *IT.java, *ITCase.java), so unit and integration tests are selected by filename. No tags, no profiles.
2. It binds to two phases. integration-test runs the tests but records the result instead of failing immediately; verify reads that record and fails the build. The gap between them is where you'd start and stop a server — teardown still runs even when a test failed, so nothing is left dangling.

Bug 1: the cache only worked on a miss

I wired failsafe, ran mvn verify, and the end-to-end journey failed on the hop where an order gets read twice.

Day 11 added Redis caching. @Cacheable on the read, @CacheEvict on the write, values serialised as JSON. The config looked like this:

ObjectMapper mapper = new ObjectMapper()
        .registerModule(new JavaTimeModule())
        .registerModule(new ParameterNamesModule());
mapper.setVisibility(PropertyAccessor.FIELD, JsonAutoDetect.Visibility.ANY);

return new GenericJackson2JsonRedisSerializer(mapper);

There's a comment above it, written on Day 11, explaining that this serializer embeds the Java type as an @class field so the value can be read back as the right type.

That comment was true — of the default constructor. GenericJackson2JsonRedisSerializer activates default typing on a mapper it builds itself. Hand it your own mapper, and it uses it exactly as given. A plain mapper writes no type information at all.

So every cached value went into Redis as anonymous JSON. Coming back out, Jackson had nothing to bind it to and produced a LinkedHashMap, which the caching proxy handed back where an Order was expected.

A cache miss worked perfectly — the method ran, returned a real Order, wrote to Redis, everything fine. A cache hit threw ClassCastException and returned a 500.

Reading the same order twice in a row. The single most ordinary thing a user can do.

Two hundred and two unit tests never saw it, and couldn't have: unit tests mock the cache away, which is the correct thing for a unit test to do. The tests that use a real Redis would have caught it on day one. They had simply never run.

The fix is to give the mapper the typing the serializer would otherwise have added — but not blindly. Unrestricted default typing is the classic Java deserialization gadget hole, so it goes through an allow-list:

PolymorphicTypeValidator typeValidator = BasicPolymorphicTypeValidator.builder()
        .allowIfSubType("dev.dev48v.orderhub.")
        .allowIfSubType("java.util.")
        .allowIfSubType("java.time.")
        .allowIfSubType("java.lang.")
        .build();

mapper.activateDefaultTyping(typeValidator, ObjectMapper.DefaultTyping.NON_FINAL,
        JsonTypeInfo.As.PROPERTY);

NON_FINAL writes the type only where it's genuinely ambiguous. Order and the list wrapping it get an @class; String and Instant don't need one, because their field declaration already pins them.

Bug 2: every integration test run ended by being killed

With the ITs finally running, each one finished in a spray of connection-refused stack traces and this:

[ERROR] Surefire is going to kill self fork JVM.
        The exit has elapsed 30 seconds after System.exit(0).

Inside a BUILD SUCCESS.

This is a teardown-order trap, and it will catch anyone combining Testcontainers with Spring's test framework. @Container hands the containers to the JUnit extension, which stops them in an afterAll callback. But the Spring context is not closed there — the test framework caches contexts (that's the whole point, so the next class can reuse one) and closes them at JVM shutdown.

Between those two moments, the application is still running with its database pulled out from under it.

My Day 30 outbox relay is a @Scheduled method that polls for unsent rows. It kept polling into the void. Hikari sat on a dead socket for its full 30-second timeout, over and over, and the fork took so long to die that surefire gave up and killed it.

The fix is the singleton-container pattern — start them in a plain static block instead:

static final PostgreSQLContainer<?> POSTGRES = new PostgreSQLContainer<>("postgres:16-alpine");
static final GenericContainer<?> REDIS =
        new GenericContainer<>(DockerImageName.parse("redis:7-alpine")).withExposedPorts(6379);

static {
    POSTGRES.start();
    REDIS.start();
}

No @Testcontainers, no @Container. The containers now outlive every Spring context, and Testcontainers' Ryuk sidecar reaps them when the JVM exits. Same isolation, correct order, clean exit.

There's a bonus: container startup is now paid once per fork instead of once per test class. RateLimitIT went from 12.4s to 4.3s.

Bug 3: the tests were wired to run twice

The obvious way to add failsafe is to declare it with an execution of your own:

<execution>
  <id>integration-tests</id>
  <goals>
    <goal>integration-test</goal>
    <goal>verify</goal>
  </goals>
</execution>

That gave me four failsafe goals per module in the build log instead of two.

spring-boot-starter-parent already manages a failsafe execution. It has no <id>, so Maven names it default. Simply declaring the plugin is what activates it. Adding an execution under any other id appends a second one — and every integration test runs twice.

Rename the id and it merges instead:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <executions>
    <execution>
      <id>default</id>
      <goals>
        <goal>integration-test</goal>
        <goal>verify</goal>
      </goals>
    </execution>
  </executions>
</plugin>

No version and no configuration, either — the parent supplies both, including pointing failsafe at target/classes so it doesn't try to scan the fat jar Boot repackages during package.

This one is only visible in the build log, which is exactly the kind of thing that survives in a codebase for years.

The capstone itself

With the plumbing honest, the finale is a single test that boots the real application against a real PostgreSQL, a real Redis and an in-process Kafka broker, turns on every feature flag at once for the first time, and walks one order through twelve hops — asserting at each one that the feature built on that day is genuinely in the path.

Every other test in the repo proves one feature in isolation. That's right while you're building it, but it is not proof that the system works. Fifty green unit tests can pass in a codebase whose parts have quietly stopped fitting together.

Each hop appends to a journal that prints at the end, so the test doubles as the demo:

  Day 4 + 5        blank customer, quantity 0 -> rejected      400 Validation failed, no row written
  Day 7            quantity 200 over configured ceiling        400 "quantity must be at most 50"
  Day 1 + 18       order placed, stock reserved over Feign     201 b6e6f07b-..., Location /api/orders/...
  Day 2 + 3        row read straight out of PostgreSQL         image postgres:16-alpine, migrations [1, 2, 3]
  Day 11 + 12      2nd GET served from Redis, not Postgres     key 'order::b6e6f07b-...' present, TTL 600s
  Day 25 + 30      outbox row relayed to Kafka, marked sent    order-placed@0, key=b6e6f07b-...
  Day 26 + 27 + 28 saga saw both facts -> PLACED to SHIPPED    OrderShipped emitted on order-shipped@0
  Day 28           payment declined -> compensating cancel     order 10bbac67-... CANCELLED, PAYMENT_DECLINED
  Day 6            filtered + paged query finds it             status=SHIPPED page 0/1, 1 total
  Day 49           found by meaning, not keywords              "typing gear..." -> score 0.220
  Day 37           counters, timer and gauge all moved         orders_placed_total{outcome="success"} 2.0
  Day 16           duplicate POST replayed, not re-executed    1 row in orders, Idempotency-Replayed: true

One command produces that:

./capstone-demo.sh

The only thing mocked is inventory-service, because it genuinely lives in another process. Everything else is the shipping code.

The seam it found that I didn't fix

Hop 7 has a comment in it that I nearly deleted.

The saga writes the shipped status through the repository — which sits behind the read cache that hop 5 just warmed. So for a moment, the API returns a stale order while the database has the new one. The test asserts against the database, then evicts the stale entry and re-reads.

I could have hidden that by asserting only the parts that agreed. Instead the test documents it, and so does the architecture write-up. The fix is to route saga writes through an evicting service method rather than straight at the repository.

A capstone that only shows the parts that worked is a brochure, not a system.

What fifty days actually produced

surefire  202 tests   0 failures 0 errors   (unit + slice)
failsafe   28 tests   0 failures 0 errors   (integration, Docker)
BUILD SUCCESS

The 28 are the 16 that had been dormant since Day 9, plus the 12 hops of the journey.

The number I'd actually put on the wall, though, is this one: the day I made the build tell me the truth, it immediately told me three things I didn't know. Two of them were bugs a user would have hit on their second click.

If you have *IT.java files in a repo somewhere, go and check that anything is running them. It takes one plugin declaration to find out, and you might not like the answer.

Full architecture write-up and all fifty days mapped onto the final system: github.com/dev48v/order-hub-from-zero