Security Operations Platform
Project Summary (6W Pitch Framework)
- Who (Target Customers and Core Roles): Designed for security operations teams at large enterprises, with an AI full-stack engineer leading the core architecture design and business-domain development.
- What (Core Product and Problems Addressed): An integrated enterprise-grade security operations management platform that addresses delayed processing of high-volume security alerts and fragmented risk management. It provides unified capabilities for cybersecurity monitoring, asset management, vulnerability remediation, and AI-powered intelligent analysis.
- Why (Core Differentiators and Competitive Advantages): Implements a three-tier data delivery strategy combining scheduled tasks, Redis caching, and real-time aggregation with cache backfilling on misses, reducing core API response times to under 50 ms. It enables a fully automated closed loop from risk identification to protection response.
- Where (Use Cases and Deployment Environments): Supports multi-scenario enterprise security operations, firewall integration, and domain security monitoring. It also provides Docker-based container orchestration for isolated multi-environment deployments and dynamic switching between data sources.
- When (Timeliness and Real-Time Capabilities): Processes more than 100,000 raw alerts from network security devices in real time each day. It supports five-minute scheduled aggregation and cache refresh cycles, while completing full-table aggregation across more than 150 tables containing tens of thousands of records within three seconds.
- How (Technical Approach and Delivery Model): Uses a microservices architecture to separate business modules and reuse shared capability libraries. It establishes a real-time analysis and automated device-response pipeline powered by Kafka, InfluxDB, a rules engine.
Business
- AI: Integrates AI Agents to perform intelligent attack-path analysis, three-dimensional risk assessment across systems, assets, and vulnerabilities, and automated generation of remediation recommendations.
- Automation: Provides a public IP blacklist analysis API that enables firewall devices to retrieve data, creating a fully automated process from risk identification to protection response.
- Concurrency: Supports real-time processing across more than 150 business tables and over 100,000 alerts per day. Core monitoring API response times were reduced from over two seconds to under 50 ms, while Redis-only APIs support more than 500 queries per second.
- Availability: Automatically falls back to database queries when the cache becomes unavailable, ensuring 99.9% system availability.
- Localization: Enables seamless switching between the Dameng database and the PostgreSQL.
- Efficiency: Improves development and delivery efficiency through independent microservice iteration and reuse of shared capability libraries.
Tech Stack
- Backend: Spring Boot 3.2.6
- Frontend: Vue 3
- Database: InfluxDB
- Cache: Redis
- Middleware: Kafka
- Containerization: Docker
Availability Is a Load-Transfer Problem
Enterprise systems often describe cache fallback in a single sentence: when Redis is unavailable, query the database. The sentence is correct and dangerously incomplete. A cache does not merely accelerate individual requests. It absorbs repeated demand that would otherwise reach the authoritative store. When the cache fails, that demand does not disappear. It moves.
This is the central engineering truth behind resilient fallback. A cache outage is a load-transfer event. If the application normally processes 500 Redis-backed queries per second with a 99 percent hit rate, the database may ordinarily receive only five fallback queries per second from that traffic class. If Redis becomes unavailable and every request is redirected without control, the database can suddenly receive all 500 queries per second. The workload has increased one hundredfold in seconds, often while users and clients are retrying because latency is rising.
A system can therefore survive the loss of Redis and still fail as a service. The application may remain alive, health checks may remain green, and every component may behave according to its local configuration. Yet the database connection pool fills, query latency grows, request threads wait, retries multiply, and write operations compete with emergency reads. The fallback intended to preserve availability becomes the mechanism that spreads the outage.
The correct design begins from a different premise. Database fallback is a scarce emergency capability. It must be budgeted, prioritized, bounded, observable, and tested under failure. It should preserve the most important business outcomes without pretending that the database can reproduce the full throughput and latency of the cache path.
For a security operations platform, this distinction matters. Critical alert summaries, active containment status, and current risk decisions may remain essential during a cache incident. Historical charts, low-priority exports, and broad ad hoc searches may safely degrade. Availability does not mean every function continues at full quality. It means the business continues making sound security decisions while the system protects the authoritative data layer and recovers predictably.
Why the Cache Exists
Redis is frequently introduced as a latency optimization. In a mature high-concurrency platform, it also serves as a capacity boundary.
A scheduled process may aggregate data across more than 150 business tables every five minutes. The result is written once and read hundreds or thousands of times. Redis converts repeated computational work into a reusable serving asset. It protects PostgreSQL or Dameng from identical dashboard requests, absorbs burst traffic, and creates stable response times even when the source data volume grows.
This means cache capacity and database capacity are intentionally different. Redis may be sized to serve hundreds or thousands of lightweight reads per second. The database may be sized for ingestion, updates, authoritative queries, scheduled aggregation, and a bounded amount of fallback. Designing both layers for the same peak read load would remove some efficiency gained by caching and increase cost substantially.
Fallback must respect this asymmetry. The database is not an undeclared replica of the cache serving tier. It is the system of record with different latency, concurrency, and workload characteristics.
The Failure Modes Are Broader Than a Complete Redis Outage
Applications should not reduce cache health to a binary state. Redis can fail in several ways, each requiring a slightly different response.
A connection failure is obvious. The application cannot establish a TCP or TLS connection, DNS resolution fails, a network route is broken, or the Redis process is unavailable.
A command timeout is less obvious. Redis accepts the connection but commands exceed the application's latency budget because of CPU pressure, memory pressure, persistence activity, network congestion, a hot key, failover, or client-side connection exhaustion.
A partial outage may affect one node, shard, availability zone, or subset of keys. Some commands succeed while others fail. Repeated retries can make this condition worse.
A logical outage occurs when Redis is reachable but the expected keys are missing, incompatible, expired together, or unreadable after a serialization change. The infrastructure appears healthy while the application experiences a miss storm.
A stale-data condition occurs when scheduled refresh has stopped but old keys remain available. The cache is technically up, yet its business output is no longer within the freshness objective.
A failover event may briefly interrupt connections and then recover. Applications need to reconnect without generating a synchronized surge of database traffic.
Operational mistakes such as broad key deletion, FLUSHALL, an incorrect eviction policy, or a low memory limit can create the same load-transfer behavior as infrastructure failure.
The resilience design should therefore react to observed command success, latency, miss rate, and data freshness rather than rely only on Redis process health.
The Business Contract for Degraded Operation
Before implementing fallback, the enterprise must decide which functions remain available and at what quality.
A critical monitoring endpoint may be permitted to query the database under a strict concurrency cap. A current containment status endpoint may read directly from its transactional table. A historical dashboard may return the last known cached result with a visible timestamp. A large export may be temporarily rejected with HTTP 503. A low-value widget may be omitted from the page.
This is graceful degradation. It is not an implementation failure. It is an explicit decision to preserve high-value functions while protecting shared resources.
Each endpoint needs a fallback class. A practical classification includes critical, important, and deferrable. Critical endpoints receive reserved fallback permits. Important endpoints may receive stale data or a smaller result. Deferrable endpoints fail fast or move to asynchronous processing.
Each class should also define a maximum stale age. A five-minute-old alert summary may remain useful. A two-hour-old automated-containment status may not. Stale data should include generatedAt and dataThrough metadata so operators can judge it correctly.
The platform should communicate degraded mode clearly. A fast response containing old data must not appear indistinguishable from a current response. Trust depends on the system being honest about what it knows.
Fast Detection Without Long Request Delays
Redis belongs on the fast path, so its timeout must be short. An application promising sub-50-millisecond cache responses should not wait two or five seconds before deciding Redis is unavailable.
Spring Data Redis commonly uses Lettuce. The client should have explicit connection and command timeouts. In a local or low-latency data-center environment, a command timeout between 100 and 500 milliseconds may be an appropriate starting range for fallback detection, but the value must be validated against normal tail latency and failover behavior. A timeout that is too short creates false failures. A timeout that is too long consumes the entire request budget.
Retries inside one user request should be conservative. Retrying the same Redis command several times during an outage delays fallback and multiplies traffic against an impaired cluster. One fast retry may be justified for a transient connection reset if the remaining deadline permits it, but unbounded retries are harmful.
A circuit breaker prevents every request from repeatedly discovering the same outage. Resilience4j can open the cache circuit when the failure rate or slow-call rate exceeds a threshold. While open, requests skip Redis and apply the degraded-mode policy immediately. After a controlled wait, a small number of half-open probes test whether Redis has recovered.
The circuit breaker should classify only relevant failures. A cache miss is not a Redis failure. A serialization defect is not always transient. A client validation error should not open the infrastructure circuit. Metrics must distinguish misses, connection failures, command timeouts, deserialization errors, and slow calls.
Detection also needs a miss-storm threshold. Redis may be healthy while a large namespace has disappeared. A rapid increase in miss ratio can activate the same fallback protections as a cache outage. Otherwise the database receives the full load while infrastructure monitoring reports Redis as healthy.
Fallback Concurrency Must Be Bounded
The most important protection is a bulkhead between cache failure and the database.
A semaphore can limit the number of concurrent fallback aggregations. If the database can safely support 20 emergency read queries in addition to normal writes and scheduled work, the application should not permit 200. Requests that cannot obtain a permit within a short interval should receive stale data, a reduced response, or HTTP 503 according to their fallback class.
The limit should be derived from database capacity. Suppose PostgreSQL has 100 usable connections. Transactional writes and normal authoritative reads require 50. Scheduled jobs may use 10. Administrative and recovery work need 10. The fallback budget should not exceed the remaining 30, and a lower value such as 15 or 20 may be prudent to preserve headroom.
The application should enforce the budget globally and, where necessary, by tenant. Without tenant limits, one large customer or malfunctioning client can consume all fallback permits. Weighted permits can reserve capacity for critical endpoints.
Connection-pool limits are not a sufficient bulkhead. When HikariCP is exhausted, request threads wait for connection timeout, often creating long latency and memory pressure. The application should reject excess fallback work before it enters the pool.
Queue length must also be bounded. A semaphore with an unbounded waiting queue merely moves the problem. A short acquisition timeout, such as 50 or 100 milliseconds for an interactive endpoint, allows the service to fail predictably. The exact number should follow the endpoint deadline and user experience.
Single-Flight Prevents Duplicate Emergency Work
When one popular key is missing, many requests can arrive simultaneously. Even under a fallback concurrency cap, permitting several identical database aggregations wastes scarce emergency capacity.
Single-flight coordination allows one request to compute a result for a key. Other requests wait briefly, consume stale data, or fail according to policy. After calculation, the owner writes the result to Redis if available and publishes it to a local emergency cache. Waiting callers reuse the result.
In one Spring Boot process, a ConcurrentHashMap<String, CompletableFuture<Result>> or per-key lock can coordinate requests. The entry must be removed after completion to prevent memory growth. Timeouts and exceptions must complete waiting futures so callers do not remain blocked.
Across multiple instances, local single-flight is not enough. A Redis-based distributed lease cannot help when Redis itself is unavailable. Alternatives include database advisory locks, a dedicated coordination service, or accepting one calculation per application instance while relying on the global database bulkhead. The last option is often simpler and sufficiently safe if the number of instances is known and the fallback budget accounts for it.
The coordination design must avoid making the fallback path more complex than the availability value it creates. A sophisticated distributed lock can introduce another failure mode. Bounded duplicate work may be safer than fragile global coordination.
Stale-While-Revalidate as the First Degraded Path
The best fallback query is often the one the system does not execute.
The application can retain a stale copy beyond the normal freshness interval while keeping a hard maximum age. The value is fresh before the soft expiry. Between soft and hard expiry, it may be served with a stale indicator while the system attempts one controlled refresh. After hard expiry, it is no longer used unless an emergency policy explicitly allows it.
For example, an operational summary may have a five-minute soft expiry and a 30-minute hard expiry. During a short Redis or database incident, the platform can continue displaying the last known result with dataThrough visible. Critical actions should not rely on stale summaries if they require current transactional state.
A local in-memory last-known-good cache can protect against complete Redis unavailability. Caffeine provides size-bounded, time-bounded in-process caching. The local cache should contain only the highest-value summaries, have strict memory limits, and preserve freshness metadata. It should not become an invisible second Redis.
The local cache is particularly useful during Redis failover because application instances can continue serving recently accessed values. Its limits must be understood. It is not shared across instances, disappears on restart, and can contain different versions on different nodes.
Database Queries Must Be Designed for Emergency Use
Fallback should not execute the most expensive unrestricted query merely because Redis is unavailable.
An emergency query can return a reduced but useful result. It may use a shorter time window, fewer dimensions, a pre-aggregated table, or a materialized view. It may omit large lists and return only critical counts. The response can state that it is a degraded view.
Queries need statement timeouts. PostgreSQL supports statement_timeout, which can be set per transaction or connection. A fallback query for an interactive endpoint might receive 500 milliseconds or one second, depending on the business objective and measured data. A query exceeding its budget should be cancelled rather than continue consuming resources after the user request has timed out.
Fallback reads should use an isolated read pool where practical. Spring Boot can configure a separate DataSource with a small maximum pool. This prevents emergency monitoring reads from consuming every connection used by writes and core workflows.
A read replica can absorb fallback traffic, but replication lag must be visible. The response should expose the source watermark or lag if freshness matters. A replica is not helpful if it is already overloaded by scheduled aggregation or if a regional network fault affects both Redis and the replica route.
Indexes, partition pruning, and summary tables remain essential. Caching cannot compensate for an emergency query that scans billions of rows. The fallback path should be benchmarked independently and kept intentionally small.
Protecting Writes and Authoritative Work
During a cache incident, preserving database writes may be more important than serving every dashboard read. Security alerts must continue to be ingested. Response decisions must be persisted. Vulnerability status changes and audit records must remain durable.
Resource governance should reflect this priority. Separate connection pools, database roles, or replicas can isolate monitoring reads from writes. PostgreSQL workload controls are less extensive than some enterprise databases, so application-level bulkheads are especially important.
Scheduled aggregation should pause or reduce concurrency during severe fallback load. There is little value in running a broad five-minute refresh against an overloaded database while the cache is unavailable and cannot accept the result. A scheduler can check degraded-mode state and defer noncritical jobs.
Batch operations, exports, and historical reconciliation can also pause. The system should free capacity for immediate security operations and recovery.
Cache Recovery Requires Controlled Rewarming
Redis recovery does not end the incident. An empty or partially recovered cache creates another dangerous transition.
If every request immediately resumes querying Redis and missing keys fall through to the database, the application can generate a database surge while repopulating the cache. This is a cold-cache storm.
Recovery should have phases. First, Redis command health is verified through half-open probes. Second, the platform warms the highest-value keys using a controlled job. Third, read traffic gradually returns to Redis. Low-priority keys can be populated on demand under the fallback budget.
Warm-up jobs should use jitter and bounded concurrency. They should not issue thousands of database aggregations at once. Priority should follow business value and request frequency.
Circuit-breaker recovery should be gradual. A half-open state permits only a small number of Redis calls. Once success and latency remain stable, the circuit closes. If failures return, it reopens without making every request discover the problem.
The application should not erase its local last-known-good cache immediately when Redis returns. Keeping it for a short overlap period provides protection if the cluster becomes unstable again.
Retry Storms and Client Behavior
Server fallback can still fail if clients retry aggressively.
HTTP clients should retry only idempotent operations and only for transient failures. Exponential backoff with jitter is essential. A response should include Retry-After when the service is rejecting traffic temporarily.
The user interface should not poll every widget independently at short intervals. A page that issues 20 requests every five seconds creates four requests per second per user. At 250 concurrent users, that is 1,000 QPS before retries. Consolidated endpoints, conditional requests, or event-driven updates can reduce this demand.
Automated clients should respect rate limits. Tenant-level token buckets can prevent one integration from consuming the fallback budget. HTTP 429 is appropriate when the caller exceeds its allocation. HTTP 503 is appropriate when the service cannot safely complete the request because a dependency is impaired.
The service should avoid returning long timeouts that encourage synchronized client retries. Fast, explicit overload responses preserve capacity and give clients a clear signal.
Observability for Degraded Mode
A cache incident should be visible through business and technical measures.
The platform should record Redis command success, timeout, connection failure, and slow-call rate. It should track cache hits, misses, stale hits, local-cache hits, and deserialization errors. It should expose circuit-breaker state and transitions.
Fallback metrics should include attempts, permits in use, permit acquisition time, rejected requests, database duration, statement cancellations, and results by endpoint class. HikariCP pending connections and timeout counts reveal whether fallback is reaching the pool boundary.
Data freshness is a first-class metric. The difference between current time and dataThrough should be measured by endpoint. A service can be technically available while serving information that is too old for the business.
Recovery metrics should include warm-up queue length, keys restored, warm-up failure, Redis latency after recovery, and the percentage of traffic returned to the cache path.
Distributed tracing should identify REDIS, LOCAL_STALE, DATABASE_FALLBACK, or REJECTED as the delivery source. Engineers investigating an incident can then see how demand moved and which protection activated.
Alerts should avoid triggering only on Redis infrastructure health. A miss ratio spike, fallback saturation, stale-age breach, or database pool wait may identify business impact sooner.
Failure Testing as an Engineering Discipline
Fallback logic that has never operated under realistic failure is an assumption, not a capability.
Development and staging tests should stop Redis, inject command latency, remove namespaces, force key expiry, and simulate intermittent connection resets. Toxiproxy can add latency, timeouts, and connection cuts between the application and Redis. Docker Compose can stop and restart the Redis container. A controlled script can remove a key namespace to reproduce a logical outage.
The test should measure the entire system. It should observe Redis detection time, circuit opening, fallback concurrency, database CPU, connection-pool wait, write latency, stale-response rate, error rate, and recovery behavior.
The most important scenario is not one request during outage. It is sustained normal traffic at the moment Redis fails. Another important scenario is recovery with an empty cache. The platform must survive both load transfer and rewarming.
Chaos tests should begin with low traffic and safe environments. The purpose is to validate limits, not to create uncontrolled impact. The same scripts can become repeatable engineering exercises.
A Spring Boot Reference Pattern
A practical Spring Boot implementation combines several mechanisms.
Lettuce uses explicit short command and connection timeouts. Resilience4j wraps cache access with a circuit breaker and time limiter. Caffeine stores a bounded last-known-good result in the application. A semaphore or Resilience4j bulkhead limits database fallback. A per-key single-flight map prevents duplicate aggregation. A dedicated fallback DataSource has a small HikariCP pool. PostgreSQL transactions apply a statement timeout. Micrometer reports every path.
The read sequence begins by checking the Redis circuit. If the circuit is closed, the application attempts Redis. A valid value returns immediately. A miss moves to controlled single-flight aggregation. A timeout or connection failure opens or contributes to the circuit and moves to degraded policy.
The degraded policy first checks the local last-known-good cache. If the value is within the permitted stale age, it returns with stale metadata. If no acceptable stale value exists and the endpoint is permitted to use database fallback, the service attempts to acquire a bulkhead permit. If it succeeds, it executes the bounded emergency query, updates the local cache, attempts Redis backfill if available, and returns. If no permit is available, it fails fast or returns a reduced result.
This sequence protects the database while continuing to serve useful information. It also makes each path measurable.
Meaningful Availability
A claim of 99.9 percent availability means no more than approximately 43 minutes and 50 seconds of unavailability in a 30.44-day month if measured continuously. The calculation is simple. The definition is not.
The enterprise must define whether a stale response counts as available, whether a degraded endpoint counts, and which user journeys are included. A health endpoint returning HTTP 200 while critical monitoring requests time out is not business availability.
A security platform may define availability around the ability to ingest alerts, show current critical risks within an agreed freshness window, and execute or record approved response actions. Secondary analytics can have different objectives.
Fallback contributes to availability only when it preserves these outcomes without damaging the database. An uncontrolled redirect of all cache traffic may improve availability for a few seconds and then cause a larger outage. Safe degradation produces a more durable result.
Capacity Economics
Resilience has a cost. A larger database, read replicas, extra Redis nodes, local caches, and engineering controls all require investment. The correct design balances the probability and impact of cache failure against business obligations.
The enterprise should not maintain database capacity equal to unlimited cache traffic if graceful degradation is acceptable. It should maintain enough emergency capacity for critical work and protect it through policy.
The fallback budget can be modeled. If one emergency aggregate consumes 50 milliseconds of database CPU and the platform permits 20 per second, the workload consumes roughly one CPU-second per second before considering I/O and contention. The actual capacity must be measured because execution time is not identical to CPU time, but the model makes assumptions visible.
The cost of stale serving is primarily decision risk. The cost of hard rejection is lost visibility. The cost of uncontrolled fallback is systemic outage. Endpoint classification allows the organization to select the lowest-risk option for each business function.
The Leadership Standard
The most important insight is that redundancy at one layer can become overload at the next. A fallback is not safe because an alternative path exists. It is safe when the alternative path has known capacity, explicit priority, bounded concurrency, reliable timeouts, and a controlled recovery process.
The enterprise should be able to answer how many database fallbacks can run concurrently, which endpoints receive them, how long queries may execute, what users see when capacity is exhausted, how stale data is labeled, and how Redis is rewarmed. If these answers are not explicit, the database is carrying an unmeasured availability promise.
A well-engineered system behaves calmly during cache failure. It detects the condition quickly. It stops wasting time on repeated Redis retries. It serves an acceptable last-known value where permitted. It reserves database capacity for high-value requests. It rejects excess work before queues become destructive. It protects writes. It warms Redis gradually after recovery. It records every transition.
This behavior reflects a broader principle of durable enterprise engineering. Systems should not rely on every component being healthy. They should decide in advance how value is preserved when capacity disappears.
Automatically falling back to database queries is therefore not a line of exception-handling code. It is an operating model for scarcity. Done poorly, it transfers failure and enlarges the incident. Done well, it protects authoritative truth, preserves essential security operations, and creates the conditions for orderly recovery.
That is the standard worth raising. Availability is not the absence of failure. It is the ability to continue making sound decisions while failure is occurring.
Top comments (0)