DEV Community

Martin D
Martin D

Posted on

Cache-Unavailable Database Fallback: Local Development Setup

Security Operations Platform

Project Summary (6W Pitch Framework)

  • Who (Target Customers and Core Roles): Designed for security operations teams at large enterprises, with an AI full-stack engineer leading the core architecture design and business-domain development.
  • What (Core Product and Problems Addressed): An integrated enterprise-grade security operations management platform that addresses delayed processing of high-volume security alerts and fragmented risk management. It provides unified capabilities for cybersecurity monitoring, asset management, vulnerability remediation, and AI-powered intelligent analysis.
  • Why (Core Differentiators and Competitive Advantages): Implements a three-tier data delivery strategy combining scheduled tasks, Redis caching, and real-time aggregation with cache backfilling on misses, reducing core API response times to under 50 ms. It enables a fully automated closed loop from risk identification to protection response.
  • Where (Use Cases and Deployment Environments): Supports multi-scenario enterprise security operations, firewall integration, and domain security monitoring. It also provides Docker-based container orchestration for isolated multi-environment deployments and dynamic switching between data sources.
  • When (Timeliness and Real-Time Capabilities): Processes more than 100,000 raw alerts from network security devices in real time each day. It supports five-minute scheduled aggregation and cache refresh cycles, while completing full-table aggregation across more than 150 tables containing tens of thousands of records within three seconds.
  • How (Technical Approach and Delivery Model): Uses a microservices architecture to separate business modules and reuse shared capability libraries. It establishes a real-time analysis and automated device-response pipeline powered by Kafka, InfluxDB, a rules engine.

Business

  • AI: Integrates AI Agents to perform intelligent attack-path analysis, three-dimensional risk assessment across systems, assets, and vulnerabilities, and automated generation of remediation recommendations.
  • Automation: Provides a public IP blacklist analysis API that enables firewall devices to retrieve data, creating a fully automated process from risk identification to protection response.
  • Concurrency: Supports real-time processing across more than 150 business tables and over 100,000 alerts per day. Core monitoring API response times were reduced from over two seconds to under 50 ms, while Redis-only APIs support more than 500 queries per second.
  • Availability: Automatically falls back to database queries when the cache becomes unavailable, ensuring 99.9% system availability.
  • Localization: Enables seamless switching between the Dameng database and the PostgreSQL.
  • Efficiency: Improves development and delivery efficiency through independent microservice iteration and reuse of shared capability libraries.

Tech Stack

  • Backend: Spring Boot 3.2.6
  • Frontend: Vue 3
  • Database: InfluxDB
  • Cache: Redis
  • Middleware: Kafka
  • Containerization: Docker

This guide builds a local Spring Boot 3.2.6 service that normally reads monitoring summaries from Redis and automatically falls back to PostgreSQL when Redis is unavailable. The fallback is deliberately protected with short Redis timeouts, a circuit breaker, a last-known-good local cache, single-flight coordination, bounded database concurrency, a dedicated fallback connection pool, query timeouts, and explicit degraded-response metadata.

The objective is not to redirect unlimited Redis traffic to PostgreSQL. The objective is to preserve important monitoring reads while preventing a cache failure from becoming a database outage.

The local environment uses Java 17, Maven, PostgreSQL 16, Redis 7.2, Spring Data JDBC, Spring Data Redis, Caffeine, Resilience4j, Micrometer, Docker Compose, Toxiproxy, curl, and k6. Toxiproxy sits between the application and Redis so latency, connection loss, and recovery can be tested without changing application code.

This guide is limited to local development. It does not include production acceptance criteria or production acceptance tests.

Step 1: Prepare the Local Development Environment

Install the following tools:

  • Java Development Kit 17
  • Apache Maven 3.9 or later
  • Docker Desktop or Docker Engine with Docker Compose v2
  • Git
  • curl
  • k6

Confirm the installations:

java -version
mvn -version
docker --version
docker compose version
git --version
curl --version
k6 version
Enter fullscreen mode Exit fullscreen mode

Create the project structure:

mkdir cache-database-fallback
cd cache-database-fallback

mkdir -p src/main/java/com/example/fallback/config
mkdir -p src/main/java/com/example/fallback/controller
mkdir -p src/main/java/com/example/fallback/dto
mkdir -p src/main/java/com/example/fallback/exception
mkdir -p src/main/java/com/example/fallback/repository
mkdir -p src/main/java/com/example/fallback/service
mkdir -p src/main/resources
mkdir -p src/test/java/com/example/fallback/service
mkdir -p load-test
Enter fullscreen mode Exit fullscreen mode

The completed request path will behave as follows. A normal request reads Redis through Toxiproxy. A successful Redis response updates an in-process Caffeine last-known-good cache. A Redis failure contributes to the circuit breaker. When the circuit is open, the application skips Redis. It first returns an acceptable local stale value. If no acceptable local value exists, it attempts a bounded PostgreSQL fallback. Excess fallback work is rejected quickly with HTTP 503 rather than waiting for the database pool to collapse.

Step 2: Start PostgreSQL, Redis, and Toxiproxy

Create compose.yaml:

services:
  postgres:
    image: postgres:16-alpine
    container_name: fallback-postgres
    environment:
      POSTGRES_DB: security_monitoring
      POSTGRES_USER: security_user
      POSTGRES_PASSWORD: security_password
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U security_user -d security_monitoring"]
      interval: 5s
      timeout: 3s
      retries: 20

  redis:
    image: redis:7.2-alpine
    container_name: fallback-redis
    command:
      - redis-server
      - --appendonly
      - "yes"
      - --maxmemory
      - 256mb
      - --maxmemory-policy
      - allkeys-lfu
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 20

  toxiproxy:
    image: ghcr.io/shopify/toxiproxy:2.9.0
    container_name: fallback-toxiproxy
    ports:
      - "8474:8474"
      - "6380:6380"
    depends_on:
      redis:
        condition: service_healthy

volumes:
  postgres_data:
  redis_data:
Enter fullscreen mode Exit fullscreen mode

Start the containers:

docker compose up -d
Enter fullscreen mode Exit fullscreen mode

Check their state:

docker compose ps
Enter fullscreen mode Exit fullscreen mode

Create a Toxiproxy proxy that listens on local port 6380 and forwards traffic to the Redis container:

curl -s -X POST http://localhost:8474/proxies \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "redis",
    "listen": "0.0.0.0:6380",
    "upstream": "redis:6379",
    "enabled": true
  }'
Enter fullscreen mode Exit fullscreen mode

Confirm the proxy:

curl -s http://localhost:8474/proxies/redis
Enter fullscreen mode Exit fullscreen mode

Test Redis through Toxiproxy:

docker run --rm --network host redis:7.2-alpine \
  redis-cli -h 127.0.0.1 -p 6380 ping
Enter fullscreen mode Exit fullscreen mode

The response should be PONG. If Docker Desktop does not support --network host, install redis-cli locally or use docker exec against the Redis container for basic connectivity. The Spring application will use localhost:6380.

Step 3: Create the Spring Boot Project

Create pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.2.6</version>
        <relativePath/>
    </parent>

    <groupId>com.example</groupId>
    <artifactId>cache-database-fallback</artifactId>
    <version>0.0.1-SNAPSHOT</version>

    <properties>
        <java.version>17</java.version>
        <resilience4j.version>2.2.0</resilience4j.version>
        <caffeine.version>3.1.8</caffeine.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-jdbc</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-redis</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>
        <dependency>
            <groupId>io.github.resilience4j</groupId>
            <artifactId>resilience4j-spring-boot3</artifactId>
            <version>${resilience4j.version}</version>
        </dependency>
        <dependency>
            <groupId>com.github.ben-manes.caffeine</groupId>
            <artifactId>caffeine</artifactId>
            <version>${caffeine.version}</version>
        </dependency>
        <dependency>
            <groupId>org.postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-validation</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>
Enter fullscreen mode Exit fullscreen mode

Create src/main/resources/application.yml:

server:
  port: 8080
  tomcat:
    threads:
      max: 100
      min-spare: 10
    accept-count: 100

spring:
  application:
    name: cache-database-fallback
  datasource:
    url: jdbc:postgresql://localhost:5432/security_monitoring
    username: security_user
    password: security_password
    hikari:
      pool-name: primary-pool
      maximum-pool-size: 12
      minimum-idle: 3
      connection-timeout: 1000
      validation-timeout: 500
  data:
    redis:
      host: localhost
      port: 6380
      connect-timeout: 250ms
      timeout: 250ms
      lettuce:
        pool:
          max-active: 16
          max-idle: 8
          min-idle: 2
          max-wait: 100ms
  sql:
    init:
      mode: always
      continue-on-error: true

management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus,circuitbreakers,circuitbreakerevents
  endpoint:
    health:
      show-details: always
  health:
    circuitbreakers:
      enabled: true
  metrics:
    distribution:
      percentiles-histogram:
        http.server.requests: true

resilience4j:
  circuitbreaker:
    instances:
      redisRead:
        register-health-indicator: true
        sliding-window-type: COUNT_BASED
        sliding-window-size: 10
        minimum-number-of-calls: 5
        failure-rate-threshold: 50
        slow-call-rate-threshold: 50
        slow-call-duration-threshold: 200ms
        wait-duration-in-open-state: 10s
        permitted-number-of-calls-in-half-open-state: 2
        automatic-transition-from-open-to-half-open-enabled: true

app:
  cache:
    prefix: monitoring:v1
    redis-ttl: 10m
    local-maximum-size: 1000
    local-hard-age: 30m
  fallback:
    maximum-concurrent-queries: 8
    permit-wait: 100ms
    query-timeout-ms: 800
    window-hours: 24
  scheduler:
    refresh-delay-ms: 300000
    tenants:
      - tenant-a
      - tenant-b

logging:
  pattern:
    level: "%5p [source:%X{deliverySource:-none}]"
  level:
    com.example.fallback: INFO
Enter fullscreen mode Exit fullscreen mode

Redis runs through Toxiproxy on port 6380. Connection and command timeouts are 250 milliseconds. The redisRead circuit opens after at least five calls when half or more fail or become slower than 200 milliseconds. It remains open for ten seconds, then permits two half-open probes.

The fallback bulkhead allows eight concurrent database queries and waits at most 100 milliseconds for a permit. Each fallback query receives an 800-millisecond PostgreSQL statement timeout.

Create src/main/java/com/example/fallback/FallbackApplication.java:

package com.example.fallback;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.scheduling.annotation.EnableScheduling;

@EnableScheduling
@SpringBootApplication
public class FallbackApplication {
    public static void main(String[] args) {
        SpringApplication.run(FallbackApplication.class, args);
    }
}
Enter fullscreen mode Exit fullscreen mode

Step 4: Create the Database Schema and Local Data

Create src/main/resources/schema.sql:

CREATE TABLE IF NOT EXISTS security_alert (
    id BIGSERIAL PRIMARY KEY,
    tenant_id VARCHAR(64) NOT NULL,
    severity VARCHAR(16) NOT NULL,
    status VARCHAR(16) NOT NULL,
    asset_id VARCHAR(128) NOT NULL,
    detected_at TIMESTAMPTZ NOT NULL
);

CREATE INDEX IF NOT EXISTS idx_alert_tenant_time
    ON security_alert (tenant_id, detected_at DESC);

CREATE INDEX IF NOT EXISTS idx_alert_tenant_status_time
    ON security_alert (tenant_id, status, detected_at DESC);
Enter fullscreen mode Exit fullscreen mode

Create src/main/resources/data.sql:

INSERT INTO security_alert
    (tenant_id, severity, status, asset_id, detected_at)
SELECT
    CASE WHEN n % 2 = 0 THEN 'tenant-a' ELSE 'tenant-b' END,
    CASE
        WHEN n % 20 = 0 THEN 'CRITICAL'
        WHEN n % 5 = 0 THEN 'HIGH'
        WHEN n % 2 = 0 THEN 'MEDIUM'
        ELSE 'LOW'
    END,
    CASE WHEN n % 4 = 0 THEN 'CLOSED' ELSE 'OPEN' END,
    'asset-' || LPAD(((n % 5000) + 1)::text, 5, '0'),
    CURRENT_TIMESTAMP - ((n % 1440) || ' minutes')::interval
FROM generate_series(1, 100000) AS n
WHERE NOT EXISTS (SELECT 1 FROM security_alert LIMIT 1);
Enter fullscreen mode Exit fullscreen mode

The seed data is synthetic and intended only for local development.

Step 5: Define the Response Contract

Create src/main/java/com/example/fallback/dto/MonitoringSummary.java:

package com.example.fallback.dto;

import java.time.Duration;
import java.time.Instant;
import java.util.Map;

public record MonitoringSummary(
    String tenantId,
    Instant windowStart,
    Instant generatedAt,
    Instant dataThrough,
    long totalAlerts,
    long openAlerts,
    Map<String, Long> severityCounts,
    String deliverySource,
    boolean degraded,
    long ageSeconds,
    int schemaVersion
) {
    public MonitoringSummary deliveredAs(String source, boolean degraded, Instant now) {
        long age = Math.max(0, Duration.between(generatedAt, now).toSeconds());
        return new MonitoringSummary(
            tenantId,
            windowStart,
            generatedAt,
            dataThrough,
            totalAlerts,
            openAlerts,
            severityCounts,
            source,
            degraded,
            age,
            schemaVersion
        );
    }
}
Enter fullscreen mode Exit fullscreen mode

The response does not hide degraded operation. It identifies the delivery source, whether the result is degraded, and how old the result is.

Step 6: Configure Redis and the Last-Known-Good Cache

Create src/main/java/com/example/fallback/config/CacheConfig.java:

package com.example.fallback.config;

import com.example.fallback.dto.MonitoringSummary;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.github.benmanes.caffeine.cache.Cache;
import com.github.benmanes.caffeine.cache.Caffeine;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.Jackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;

import java.time.Duration;

@Configuration
public class CacheConfig {

    @Bean
    RedisTemplate<String, MonitoringSummary> monitoringRedisTemplate(
        RedisConnectionFactory connectionFactory,
        ObjectMapper objectMapper
    ) {
        RedisTemplate<String, MonitoringSummary> template = new RedisTemplate<>();
        template.setConnectionFactory(connectionFactory);
        template.setKeySerializer(new StringRedisSerializer());
        template.setValueSerializer(
            new Jackson2JsonRedisSerializer<>(objectMapper, MonitoringSummary.class)
        );
        template.afterPropertiesSet();
        return template;
    }

    @Bean
    Cache<String, MonitoringSummary> lastKnownGoodCache(
        @Value("${app.cache.local-maximum-size}") long maximumSize,
        @Value("${app.cache.local-hard-age}") Duration hardAge
    ) {
        return Caffeine.newBuilder()
            .maximumSize(maximumSize)
            .expireAfterWrite(hardAge)
            .recordStats()
            .build();
    }
}
Enter fullscreen mode Exit fullscreen mode

Caffeine stores at most 1,000 summaries and keeps each value for no more than 30 minutes. It is not authoritative and is not shared between application instances. Its role is to serve a recently known result during a short Redis incident.

Step 7: Create a Dedicated Fallback Database Pool

Create src/main/java/com/example/fallback/config/FallbackDatabaseConfig.java:

package com.example.fallback.config;

import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate;

import javax.sql.DataSource;

@Configuration
public class FallbackDatabaseConfig {

    @Bean(destroyMethod = "close")
    DataSource fallbackDataSource(
        @Value("${spring.datasource.url}") String url,
        @Value("${spring.datasource.username}") String username,
        @Value("${spring.datasource.password}") String password,
        @Value("${app.fallback.query-timeout-ms}") int queryTimeoutMs
    ) {
        HikariConfig config = new HikariConfig();
        config.setPoolName("fallback-read-pool");
        config.setJdbcUrl(url);
        config.setUsername(username);
        config.setPassword(password);
        config.setMaximumPoolSize(8);
        config.setMinimumIdle(1);
        config.setConnectionTimeout(300);
        config.setValidationTimeout(250);
        config.setReadOnly(true);
        config.setConnectionInitSql(
            "SET statement_timeout = " + queryTimeoutMs
        );
        return new HikariDataSource(config);
    }

    @Bean
    NamedParameterJdbcTemplate fallbackJdbcTemplate(DataSource fallbackDataSource) {
        return new NamedParameterJdbcTemplate(fallbackDataSource);
    }
}
Enter fullscreen mode Exit fullscreen mode

The fallback pool has only eight connections and is read-only. It cannot consume the primary pool. PostgreSQL cancels statements running longer than 800 milliseconds.

Step 8: Implement the Emergency Database Query

Create src/main/java/com/example/fallback/repository/FallbackSummaryRepository.java:

package com.example.fallback.repository;

import com.example.fallback.dto.MonitoringSummary;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate;
import org.springframework.stereotype.Repository;

import java.time.Clock;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.LinkedHashMap;
import java.util.Map;

@Repository
public class FallbackSummaryRepository {

    private final NamedParameterJdbcTemplate jdbc;
    private final long windowHours;
    private final Clock clock = Clock.systemUTC();

    public FallbackSummaryRepository(
        NamedParameterJdbcTemplate fallbackJdbcTemplate,
        @Value("${app.fallback.window-hours}") long windowHours
    ) {
        this.jdbc = fallbackJdbcTemplate;
        this.windowHours = windowHours;
    }

    public MonitoringSummary aggregate(String tenantId) {
        Instant generatedAt = clock.instant();
        Instant windowStart = generatedAt.minus(windowHours, ChronoUnit.HOURS);

        MapSqlParameterSource parameters = new MapSqlParameterSource()
            .addValue("tenantId", tenantId)
            .addValue("windowStart", windowStart);

        String severitySql = """
            SELECT severity, COUNT(*) AS total
            FROM security_alert
            WHERE tenant_id = :tenantId
              AND detected_at >= :windowStart
            GROUP BY severity
            """;

        Map<String, Long> severityCounts = new LinkedHashMap<>();
        severityCounts.put("CRITICAL", 0L);
        severityCounts.put("HIGH", 0L);
        severityCounts.put("MEDIUM", 0L);
        severityCounts.put("LOW", 0L);

        jdbc.query(severitySql, parameters, rs ->
            severityCounts.put(rs.getString("severity"), rs.getLong("total"))
        );

        Long openAlerts = jdbc.queryForObject(
            """
            SELECT COUNT(*)
            FROM security_alert
            WHERE tenant_id = :tenantId
              AND status = 'OPEN'
              AND detected_at >= :windowStart
            """,
            parameters,
            Long.class
        );

        long totalAlerts = severityCounts.values()
            .stream()
            .mapToLong(Long::longValue)
            .sum();

        return new MonitoringSummary(
            tenantId,
            windowStart,
            generatedAt,
            generatedAt,
            totalAlerts,
            openAlerts == null ? 0 : openAlerts,
            Map.copyOf(severityCounts),
            "DATABASE_FALLBACK",
            true,
            0,
            1
        );
    }
}
Enter fullscreen mode Exit fullscreen mode

The emergency query returns a compact 24-hour summary. It does not return a large alert list or perform broad cross-domain joins.

Step 9: Implement Circuit-Broken Redis Access

Create src/main/java/com/example/fallback/service/RedisSummaryStore.java:

package com.example.fallback.service;

import com.example.fallback.dto.MonitoringSummary;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Service;

import java.time.Duration;
import java.util.Optional;

@Service
public class RedisSummaryStore {

    private final RedisTemplate<String, MonitoringSummary> redisTemplate;
    private final String prefix;
    private final Duration ttl;
    private final Counter hitCounter;
    private final Counter missCounter;

    public RedisSummaryStore(
        RedisTemplate<String, MonitoringSummary> redisTemplate,
        MeterRegistry registry,
        @Value("${app.cache.prefix}") String prefix,
        @Value("${app.cache.redis-ttl}") Duration ttl
    ) {
        this.redisTemplate = redisTemplate;
        this.prefix = prefix;
        this.ttl = ttl;
        this.hitCounter = registry.counter("fallback.redis.hit");
        this.missCounter = registry.counter("fallback.redis.miss");
    }

    @CircuitBreaker(name = "redisRead")
    public Optional<MonitoringSummary> get(String tenantId) {
        MonitoringSummary value = redisTemplate.opsForValue().get(key(tenantId));
        if (value == null) {
            missCounter.increment();
            return Optional.empty();
        }
        hitCounter.increment();
        return Optional.of(value);
    }

    public void put(String tenantId, MonitoringSummary summary) {
        redisTemplate.opsForValue().set(key(tenantId), summary, ttl);
    }

    public void evict(String tenantId) {
        redisTemplate.delete(key(tenantId));
    }

    public String key(String tenantId) {
        return prefix + ":tenant:" + tenantId + ":summary:24h";
    }
}
Enter fullscreen mode Exit fullscreen mode

Connection failures, command timeouts, and open-circuit errors propagate to the orchestration service. A normal cache miss is returned as Optional.empty() and does not count as a circuit-breaker failure.

Step 10: Implement Bounded Fallback and Single-Flight Coordination

Create src/main/java/com/example/fallback/exception/FallbackBusyException.java:

package com.example.fallback.exception;

public class FallbackBusyException extends RuntimeException {
    public FallbackBusyException(String message) {
        super(message);
    }
}
Enter fullscreen mode Exit fullscreen mode

Create src/main/java/com/example/fallback/service/FallbackCoordinator.java:

package com.example.fallback.service;

import com.example.fallback.dto.MonitoringSummary;
import com.example.fallback.exception.FallbackBusyException;
import com.example.fallback.repository.FallbackSummaryRepository;
import com.github.benmanes.caffeine.cache.Cache;
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.MeterRegistry;
import org.slf4j.MDC;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

import java.time.Clock;
import java.time.Duration;
import java.time.Instant;
import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock;

@Service
public class FallbackCoordinator {

    private final RedisSummaryStore redisStore;
    private final FallbackSummaryRepository repository;
    private final Cache<String, MonitoringSummary> localCache;
    private final Semaphore fallbackPermits;
    private final Duration permitWait;
    private final Duration localHardAge;
    private final ConcurrentHashMap<String, ReentrantLock> keyLocks =
        new ConcurrentHashMap<>();
    private final Clock clock = Clock.systemUTC();

    private final Counter redisCounter;
    private final Counter staleCounter;
    private final Counter databaseCounter;
    private final Counter rejectedCounter;

    public FallbackCoordinator(
        RedisSummaryStore redisStore,
        FallbackSummaryRepository repository,
        Cache<String, MonitoringSummary> lastKnownGoodCache,
        MeterRegistry registry,
        @Value("${app.fallback.maximum-concurrent-queries}") int maximumConcurrent,
        @Value("${app.fallback.permit-wait}") Duration permitWait,
        @Value("${app.cache.local-hard-age}") Duration localHardAge
    ) {
        this.redisStore = redisStore;
        this.repository = repository;
        this.localCache = lastKnownGoodCache;
        this.fallbackPermits = new Semaphore(maximumConcurrent, true);
        this.permitWait = permitWait;
        this.localHardAge = localHardAge;
        this.redisCounter = registry.counter("fallback.delivery", "source", "redis");
        this.staleCounter = registry.counter("fallback.delivery", "source", "local_stale");
        this.databaseCounter = registry.counter("fallback.delivery", "source", "database");
        this.rejectedCounter = registry.counter("fallback.delivery", "source", "rejected");
        Gauge.builder("fallback.permits.available", fallbackPermits, Semaphore::availablePermits)
            .register(registry);
    }

    public MonitoringSummary get(String tenantId) {
        Instant now = clock.instant();

        try {
            Optional<MonitoringSummary> cached = redisStore.get(tenantId);
            if (cached.isPresent()) {
                MonitoringSummary value = cached.orElseThrow();
                localCache.put(tenantId, value);
                redisCounter.increment();
                MDC.put("deliverySource", "REDIS");
                return value.deliveredAs("REDIS", false, now);
            }
        } catch (Exception ignored) {
            // The circuit breaker and metrics record Redis failure.
        }

        Optional<MonitoringSummary> stale = acceptableLocalValue(tenantId, now);
        if (stale.isPresent()) {
            staleCounter.increment();
            MDC.put("deliverySource", "LOCAL_STALE");
            return stale.orElseThrow().deliveredAs("LOCAL_STALE", true, now);
        }

        return singleFlightDatabaseFallback(tenantId);
    }

    public MonitoringSummary refresh(String tenantId) {
        MonitoringSummary summary = executeDatabaseFallback(tenantId);
        try {
            redisStore.put(tenantId, summary);
        } catch (Exception ignored) {
            // The current result remains useful even if Redis backfill fails.
        }
        localCache.put(tenantId, summary);
        return summary;
    }

    public void evictRedis(String tenantId) {
        redisStore.evict(tenantId);
    }

    public void clearLocal(String tenantId) {
        localCache.invalidate(tenantId);
    }

    private MonitoringSummary singleFlightDatabaseFallback(String tenantId) {
        ReentrantLock lock = keyLocks.computeIfAbsent(tenantId, ignored -> new ReentrantLock());
        lock.lock();
        try {
            Instant now = clock.instant();
            Optional<MonitoringSummary> becameAvailable = acceptableLocalValue(tenantId, now);
            if (becameAvailable.isPresent()) {
                staleCounter.increment();
                MDC.put("deliverySource", "LOCAL_AFTER_WAIT");
                return becameAvailable.orElseThrow()
                    .deliveredAs("LOCAL_AFTER_WAIT", true, now);
            }
            return executeDatabaseFallback(tenantId);
        } finally {
            lock.unlock();
            if (!lock.hasQueuedThreads()) {
                keyLocks.remove(tenantId, lock);
            }
        }
    }

    private MonitoringSummary executeDatabaseFallback(String tenantId) {
        boolean acquired = false;
        try {
            acquired = fallbackPermits.tryAcquire(
                permitWait.toMillis(),
                TimeUnit.MILLISECONDS
            );
            if (!acquired) {
                rejectedCounter.increment();
                MDC.put("deliverySource", "REJECTED");
                throw new FallbackBusyException(
                    "Database fallback capacity is temporarily exhausted"
                );
            }

            MonitoringSummary summary = repository.aggregate(tenantId);
            localCache.put(tenantId, summary);
            try {
                redisStore.put(tenantId, summary);
            } catch (Exception ignored) {
                // Redis remains optional during degraded operation.
            }
            databaseCounter.increment();
            MDC.put("deliverySource", "DATABASE_FALLBACK");
            return summary.deliveredAs("DATABASE_FALLBACK", true, clock.instant());
        } catch (InterruptedException ex) {
            Thread.currentThread().interrupt();
            throw new FallbackBusyException("Fallback permit wait was interrupted");
        } finally {
            if (acquired) {
                fallbackPermits.release();
            }
        }
    }

    private Optional<MonitoringSummary> acceptableLocalValue(
        String tenantId,
        Instant now
    ) {
        MonitoringSummary value = localCache.getIfPresent(tenantId);
        if (value == null) {
            return Optional.empty();
        }
        Duration age = Duration.between(value.generatedAt(), now);
        return age.compareTo(localHardAge) <= 0
            ? Optional.of(value)
            : Optional.empty();
    }
}
Enter fullscreen mode Exit fullscreen mode

The local stale path is intentionally checked before PostgreSQL. This keeps the database fallback budget available for keys without an acceptable last-known result. The per-key lock prevents repeated fallback calculations for one tenant inside one application process.

Step 11: Add Scheduled Refresh and Controlled Rewarming

Create src/main/java/com/example/fallback/service/SummaryRefreshScheduler.java:

package com.example.fallback.service;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

import java.util.List;

@Component
public class SummaryRefreshScheduler {

    private static final Logger log = LoggerFactory.getLogger(SummaryRefreshScheduler.class);

    private final FallbackCoordinator coordinator;
    private final List<String> tenants;

    public SummaryRefreshScheduler(
        FallbackCoordinator coordinator,
        @Value("${app.scheduler.tenants}") List<String> tenants
    ) {
        this.coordinator = coordinator;
        this.tenants = List.copyOf(tenants);
    }

    @Scheduled(
        initialDelayString = "10000",
        fixedDelayString = "${app.scheduler.refresh-delay-ms}"
    )
    public void refresh() {
        for (String tenantId : tenants) {
            try {
                coordinator.refresh(tenantId);
                log.info("Refreshed summary for tenant {}", tenantId);
                Thread.sleep(100);
            } catch (Exception ex) {
                log.warn("Refresh failed for tenant {}: {}", tenantId, ex.getMessage());
            }
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

The 100-millisecond pause is a simple local warm-up control. A larger implementation should use a bounded executor, priority queue, and jitter rather than refreshing every tenant simultaneously.

Step 12: Expose the API and Degraded Errors

Create src/main/java/com/example/fallback/controller/MonitoringController.java:

package com.example.fallback.controller;

import com.example.fallback.dto.MonitoringSummary;
import com.example.fallback.service.FallbackCoordinator;
import jakarta.validation.constraints.Pattern;
import org.springframework.http.ResponseEntity;
import org.springframework.validation.annotation.Validated;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@Validated
@RestController
@RequestMapping("/api/v1/tenants/{tenantId}/monitoring-summary")
public class MonitoringController {

    private final FallbackCoordinator coordinator;

    public MonitoringController(FallbackCoordinator coordinator) {
        this.coordinator = coordinator;
    }

    @GetMapping
    public MonitoringSummary get(
        @PathVariable
        @Pattern(regexp = "[a-zA-Z0-9-]{1,64}")
        String tenantId
    ) {
        return coordinator.get(tenantId);
    }

    @PostMapping("/refresh")
    public MonitoringSummary refresh(@PathVariable String tenantId) {
        return coordinator.refresh(tenantId);
    }

    @DeleteMapping("/redis")
    public ResponseEntity<Void> evictRedis(@PathVariable String tenantId) {
        coordinator.evictRedis(tenantId);
        return ResponseEntity.noContent().build();
    }

    @DeleteMapping("/local")
    public ResponseEntity<Void> clearLocal(@PathVariable String tenantId) {
        coordinator.clearLocal(tenantId);
        return ResponseEntity.noContent().build();
    }
}
Enter fullscreen mode Exit fullscreen mode

Create src/main/java/com/example/fallback/controller/ApiExceptionHandler.java:

package com.example.fallback.controller;

import com.example.fallback.exception.FallbackBusyException;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;

import java.time.Instant;
import java.util.Map;

@RestControllerAdvice
public class ApiExceptionHandler {

    @ExceptionHandler(FallbackBusyException.class)
    ResponseEntity<Map<String, Object>> fallbackBusy(FallbackBusyException ex) {
        return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
            .header(HttpHeaders.RETRY_AFTER, "2")
            .body(Map.of(
                "code", "DATABASE_FALLBACK_BUSY",
                "message", ex.getMessage(),
                "timestamp", Instant.now().toString()
            ));
    }
}
Enter fullscreen mode Exit fullscreen mode

HTTP 503 communicates that the service is protecting an impaired dependency. Retry-After: 2 tells clients not to retry immediately.

Step 13: Build and Start the Application

Build the project:

mvn clean test
Enter fullscreen mode Exit fullscreen mode

Start the application:

MAVEN_OPTS='-Xms512m -Xmx512m' mvn spring-boot:run
Enter fullscreen mode Exit fullscreen mode

Check health:

curl -s http://localhost:8080/actuator/health
Enter fullscreen mode Exit fullscreen mode

Wait at least ten seconds for scheduled refresh. Request a summary:

curl -s http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary
Enter fullscreen mode Exit fullscreen mode

The normal response should report deliverySource as REDIS, degraded as false, and a small ageSeconds value.

Inspect the Redis key through Toxiproxy using the application endpoint rather than a direct redis-cli command. The key name is:

monitoring:v1:tenant:tenant-a:summary:24h
Enter fullscreen mode Exit fullscreen mode

Step 14: Verify a Cache Miss with Redis Available

Delete only the Redis value:

curl -i -X DELETE \
  http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary/redis
Enter fullscreen mode Exit fullscreen mode

Do not clear the local cache yet. Request the summary:

curl -s http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary
Enter fullscreen mode Exit fullscreen mode

The response should report LOCAL_STALE. Redis is available but the key is missing, so the service reuses the last-known-good local value instead of executing another database query.

Force a complete cold miss by clearing both Redis and the local cache:

curl -s -X DELETE \
  http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary/redis

curl -s -X DELETE \
  http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary/local
Enter fullscreen mode Exit fullscreen mode

Request the summary again:

curl -s http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary
Enter fullscreen mode Exit fullscreen mode

The response should report DATABASE_FALLBACK. The result is written back to Redis and Caffeine. The next request should return REDIS.

Step 15: Simulate Redis Latency

Add 500 milliseconds of latency through Toxiproxy:

curl -s -X POST \
  http://localhost:8474/proxies/redis/toxics \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "redis_latency",
    "type": "latency",
    "stream": "downstream",
    "toxicity": 1.0,
    "attributes": {
      "latency": 500,
      "jitter": 25
    }
  }'
Enter fullscreen mode Exit fullscreen mode

Send several requests:

for i in $(seq 1 10); do
  curl -s -o /dev/null -w '%{http_code} %{time_total}\n' \
    http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary
done
Enter fullscreen mode Exit fullscreen mode

The Redis command timeout is 250 milliseconds. After enough slow calls, the circuit breaker should open and later requests should skip Redis. Because the local cache contains the last successful value, responses should use LOCAL_STALE rather than PostgreSQL.

Inspect circuit-breaker health:

curl -s http://localhost:8080/actuator/health
Enter fullscreen mode Exit fullscreen mode

Inspect circuit-breaker events:

curl -s http://localhost:8080/actuator/circuitbreakerevents
Enter fullscreen mode Exit fullscreen mode

Remove the latency toxic:

curl -s -X DELETE \
  http://localhost:8474/proxies/redis/toxics/redis_latency
Enter fullscreen mode Exit fullscreen mode

Wait at least ten seconds for the open-state duration, then make two or more requests so half-open probes can verify Redis recovery.

Step 16: Simulate a Complete Redis Connection Failure

Disable the Toxiproxy proxy:

curl -s -X POST http://localhost:8474/proxies/redis \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "redis",
    "listen": "0.0.0.0:6380",
    "upstream": "redis:6379",
    "enabled": false
  }'
Enter fullscreen mode Exit fullscreen mode

Request the warmed tenant:

curl -s http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary
Enter fullscreen mode Exit fullscreen mode

The expected delivery source is LOCAL_STALE because the in-process cache contains a valid last-known result.

Clear the local cache while Redis remains unavailable:

curl -s -X DELETE \
  http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary/local
Enter fullscreen mode Exit fullscreen mode

Request the summary again:

curl -s http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary
Enter fullscreen mode Exit fullscreen mode

The expected delivery source is DATABASE_FALLBACK. The service cannot backfill Redis, but it stores the result in Caffeine. A following request should use LOCAL_STALE.

Re-enable the proxy:

curl -s -X POST http://localhost:8474/proxies/redis \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "redis",
    "listen": "0.0.0.0:6380",
    "upstream": "redis:6379",
    "enabled": true
  }'
Enter fullscreen mode Exit fullscreen mode

Wait for the circuit breaker to transition to half-open. Request the summary until the circuit closes. Run a manual refresh to rewarm Redis:

curl -s -X POST \
  http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary/refresh
Enter fullscreen mode Exit fullscreen mode

Step 17: Verify Single-Flight Fallback

Disable Redis again. Clear the local values for a new tenant key:

curl -s -X POST http://localhost:8474/proxies/redis \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "redis",
    "listen": "0.0.0.0:6380",
    "upstream": "redis:6379",
    "enabled": false
  }'
Enter fullscreen mode Exit fullscreen mode

Send 20 concurrent requests for the same cold key:

seq 1 20 | xargs -n1 -P20 -I{} \
  curl -s http://localhost:8080/api/v1/tenants/cold-tenant/monitoring-summary
Enter fullscreen mode Exit fullscreen mode

One request should perform the PostgreSQL aggregation. Waiting requests should reuse the local value after the first query completes. This protects the database from 20 identical emergency queries inside one application process.

Inspect database delivery metrics:

curl -s 'http://localhost:8080/actuator/metrics/fallback.delivery?tag=source:database'
Enter fullscreen mode Exit fullscreen mode

The database delivery counter should increase once for the shared cold key under normal local timing.

Step 18: Verify the Database Bulkhead

Keep Redis disabled. Generate requests for many unique cold tenant keys so single-flight cannot combine them:

seq 1 50 | xargs -n1 -P50 -I{} \
  curl -s -o /dev/null -w '%{http_code}\n' \
  http://localhost:8080/api/v1/tenants/cold-{}/monitoring-summary
Enter fullscreen mode Exit fullscreen mode

Only eight database fallbacks can run at once. Requests unable to obtain a permit within 100 milliseconds receive HTTP 503 with DATABASE_FALLBACK_BUSY and Retry-After: 2.

Inspect the metrics:

curl -s http://localhost:8080/actuator/metrics/fallback.permits.available
curl -s 'http://localhost:8080/actuator/metrics/fallback.delivery?tag=source:rejected'
curl -s http://localhost:8080/actuator/metrics/hikaricp.connections.pending
Enter fullscreen mode Exit fullscreen mode

The important behavior is bounded pressure. The service rejects excess reads before they consume every database connection and request thread.

Step 19: Run a Development Cache-Failure Load Test

Create load-test/cache_failure.js:

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  scenarios: {
    degraded_reads: {
      executor: 'constant-arrival-rate',
      rate: 100,
      timeUnit: '1s',
      duration: '60s',
      preAllocatedVUs: 50,
      maxVUs: 200,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.05'],
    checks: ['rate>0.95'],
  },
};

export default function () {
  const response = http.get(
    'http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary'
  );

  check(response, {
    'status is 200 or protected 503': (r) =>
      r.status === 200 || r.status === 503,
    'degraded response identifies its source': (r) => {
      if (r.status === 503) return true;
      try {
        const body = JSON.parse(r.body);
        return ['LOCAL_STALE', 'LOCAL_AFTER_WAIT', 'DATABASE_FALLBACK']
          .includes(body.deliverySource);
      } catch (e) {
        return false;
      }
    },
  });

  sleep(0.01);
}
Enter fullscreen mode Exit fullscreen mode

Warm tenant-a first, then disable Redis and run the test:

k6 run load-test/cache_failure.js
Enter fullscreen mode Exit fullscreen mode

Because tenant-a has a last-known-good value, most requests should return LOCAL_STALE. This test demonstrates that the platform can avoid transferring 100 QPS to PostgreSQL during the cache outage.

For a cold-key overload test, use unique tenant names and expect controlled HTTP 503 responses once the fallback bulkhead is full. Do not interpret those responses as a defect. They demonstrate dependency protection in the development scenario.

Step 20: Observe the Degraded Paths

Inspect all delivery-source counters:

curl -s http://localhost:8080/actuator/metrics/fallback.delivery
Enter fullscreen mode Exit fullscreen mode

Inspect Redis metrics:

curl -s http://localhost:8080/actuator/metrics/fallback.redis.hit
curl -s http://localhost:8080/actuator/metrics/fallback.redis.miss
Enter fullscreen mode Exit fullscreen mode

Inspect HTTP percentiles and counts:

curl -s http://localhost:8080/actuator/metrics/http.server.requests
Enter fullscreen mode Exit fullscreen mode

Inspect Prometheus output:

curl -s http://localhost:8080/actuator/prometheus | \
  grep -E 'fallback_|resilience4j_circuitbreaker|hikaricp|http_server_requests'
Enter fullscreen mode Exit fullscreen mode

Useful local observations include the time for the circuit to open, the proportion of local stale responses, the number of database fallbacks, the number of rejected requests, fallback pool usage, and response age.

Step 21: Verify Query Timeout Behavior

Temporarily reduce the query timeout in application.yml:

app:
  fallback:
    query-timeout-ms: 1
Enter fullscreen mode Exit fullscreen mode

Restart the application, disable Redis, clear the local cache, and request a cold key. PostgreSQL should cancel the aggregation quickly. Restore the value to 800 milliseconds after observing the behavior.

The purpose is to confirm that failed client requests do not leave expensive queries running indefinitely. The exact timeout must be longer than normal query latency but shorter than the endpoint's maximum degraded response budget.

Step 22: Rewarm Redis Gradually After Recovery

Re-enable Redis through Toxiproxy. Allow the circuit breaker's ten-second open period to expire. Send a small number of requests so the two half-open calls can verify Redis health.

Refresh the highest-value tenants first:

curl -s -X POST \
  http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary/refresh

sleep 1

curl -s -X POST \
  http://localhost:8080/api/v1/tenants/tenant-b/monitoring-summary/refresh
Enter fullscreen mode Exit fullscreen mode

Do not issue thousands of refresh requests at once. A large cold-cache warm-up can overload PostgreSQL immediately after an incident. The scheduler's small pause demonstrates controlled rewarming locally.

Retain Caffeine values during the overlap. They provide continuity if Redis becomes unstable again during recovery.

Step 23: Add Focused Development Tests

Create src/test/java/com/example/fallback/service/MonitoringSummaryTest.java:

package com.example.fallback.service;

import com.example.fallback.dto.MonitoringSummary;
import org.junit.jupiter.api.Test;

import java.time.Instant;
import java.util.Map;

import static org.assertj.core.api.Assertions.assertThat;

class MonitoringSummaryTest {

    @Test
    void marksLastKnownValueAsDegraded() {
        Instant generatedAt = Instant.parse("2026-11-19T10:00:00Z");
        MonitoringSummary original = new MonitoringSummary(
            "tenant-a",
            Instant.parse("2026-11-18T10:00:00Z"),
            generatedAt,
            generatedAt,
            100,
            80,
            Map.of("CRITICAL", 5L, "HIGH", 20L),
            "REDIS",
            false,
            0,
            1
        );

        MonitoringSummary degraded = original.deliveredAs(
            "LOCAL_STALE",
            true,
            Instant.parse("2026-11-19T10:05:00Z")
        );

        assertThat(degraded.deliverySource()).isEqualTo("LOCAL_STALE");
        assertThat(degraded.degraded()).isTrue();
        assertThat(degraded.ageSeconds()).isEqualTo(300);
        assertThat(degraded.totalAlerts()).isEqualTo(100);
    }
}
Enter fullscreen mode Exit fullscreen mode

Run the tests:

mvn test
Enter fullscreen mode Exit fullscreen mode

These tests are local development tests. They are not production acceptance tests.

Step 24: Tune the Local Fallback Policy

Increase local-hard-age only when the business can safely consume older data. A longer value improves continuity but increases the possibility of acting on stale information.

Keep Redis timeouts short enough to preserve the request budget. If ordinary local p99 Redis latency is below 10 milliseconds, a 250-millisecond timeout leaves substantial failure-detection headroom. Measure before reducing it further.

Do not set the fallback semaphore higher than the dedicated pool. The sample uses eight permits and eight fallback connections so the application does not create a hidden waiting queue inside HikariCP.

Keep the permit wait short. A 100-millisecond wait allows brief contention without allowing hundreds of request threads to wait for database capacity.

Use a compact emergency query. If the normal dashboard joins many domains, its degraded database path should return only security-critical fields.

For a multi-instance local experiment, run a second application instance on another port. Remember that Caffeine and single-flight locks are process-local. The database fallback budget must account for all instances together.

Step 25: Reset the Local Environment

Re-enable the Redis proxy if it is disabled:

curl -s -X POST http://localhost:8474/proxies/redis \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "redis",
    "listen": "0.0.0.0:6380",
    "upstream": "redis:6379",
    "enabled": true
  }'
Enter fullscreen mode Exit fullscreen mode

Remove all Toxiproxy toxics:

curl -s http://localhost:8474/proxies/redis/toxics | \
  grep -o '"name":"[^"]*"' | \
  cut -d '"' -f 4 | \
  while read -r toxic; do
    curl -s -X DELETE \
      "http://localhost:8474/proxies/redis/toxics/${toxic}"
  done
Enter fullscreen mode Exit fullscreen mode

Stop the Spring Boot application with Ctrl+C.

Stop containers while preserving data:

docker compose stop
Enter fullscreen mode Exit fullscreen mode

Remove containers while preserving named volumes:

docker compose down
Enter fullscreen mode Exit fullscreen mode

Remove containers and all local data:

docker compose down -v
Enter fullscreen mode Exit fullscreen mode

Remove Maven build output:

mvn clean
Enter fullscreen mode Exit fullscreen mode

The completed local implementation demonstrates safe automatic database fallback rather than unconditional redirection. Redis remains the normal low-latency path. Resilience4j detects command failures and opens the circuit. Caffeine preserves a bounded last-known-good result. Single-flight coordination prevents duplicate emergency work for one key. A semaphore and dedicated HikariCP pool limit database pressure. PostgreSQL statement timeout cancels over-budget queries. HTTP 503 and Retry-After protect the system when fallback capacity is exhausted. Toxiproxy provides a repeatable way to test latency, outage, and recovery without changing the application code.

Top comments (0)