✓ Human-authored analysis; AI used for formatting and proofreading.
There's a class of failure where every check passes and the system is broken.
CloudTrail is enabled. Multi-region. Log file validation is on. CloudWatch Logs integration is active. The retention policy is configured. The S3 lifecycle rule exists. The compliance dashboard shows green across every line item.
The S3 delivery failed 120 days ago. The bucket policy changed or the bucket was deleted. Or the KMS key was disabled. Or the trail's IAM role lost permission to write. CloudTrail logged the failure in GetTrailStatus under LatestDeliveryError. Nobody called GetTrailStatus. Every tool checked the trail configuration. No tool checked whether the trail was actually delivering events.
Every control downstream of delivery evaluates a stale or empty bucket. The retention policy governs nothing. The lifecycle rule expires nothing. The log file validation validates nothing. The compliance report certifies a system that isn't recording.
The ghost reference in your audit trail
This is the ghost reference pattern where a configuration that appears active but references something that no longer functions. Secrets Manager rotation enabled but the rotation Lambda was deleted. An SNS topic configured as an alarm action but the topic was deleted. A CloudWatch subscription filter pointing to a destination that rejects the data.
The audit trail version is the most dangerous because it's invisible to every other security check. An S3 bucket that's public will fire a control. A trail that's not delivering won't fire anything. Because the non-delivery isn't a configuration property the trail exposes through its standard describe API. It's in a separate API call (GetTrailStatus) that most collectors don't make.
The fix is a derived boolean: s3_delivery_healthy. True when LatestDeliveryError is empty AND LatestDeliveryTime is within 24 hours. The collector makes the GetTrailStatus call, derives the boolean, stamps it on the observation. The evaluator checks the boolean. The complexity of "is delivery working?" is resolved once by the collector. The evaluator sees a field and compares it against an expected value.
What I checked
I decomposed detection blindness. The state where an attacker operates without any detection service noticing into 27 configuration properties across six AWS detection services.
CloudTrail covers the audit trail: enabled, multi-region, management events include reads (not just writes), S3 and Lambda data events, CloudWatch Logs integration, delivery health, log file validation, and retention beyond the 90-day Events History window. Seven properties.
AWS Config covers configuration drift visibility: recorder enabled, all resource types recorded, global resources included, delivery channel with S3 and SNS, and delivery health (snapshot freshness). Five properties.
GuardDuty covers threat detection: detector enabled, all protection features active (S3, EKS, Lambda, RDS, Runtime, Malware), finding export to S3, notification routing to the response team, and detection method count for cross-signal correlation. Five properties.
Security Hub covers finding aggregation: enabled, at least one standard active, auto-enable for new accounts, and cross-region aggregation. Four properties.
VPC Flow Logs cover network visibility: enabled per VPC, status ACTIVE, and custom format with security-critical fields (vpc-id, subnet-id, tcp-flags, pkt-srcaddr). Three properties.
CloudWatch and EventBridge cover the detection pipeline: log group retention policy, subscription filter destination monitoring, and CloudTrail change monitoring (CIS 4.5 — detection self-protection). Three properties.
26 of 27 covered
The existing catalog handles nearly everything. CloudTrail alone has 60 controls. Config has 51. GuardDuty has 19. The per-service coverage is thorough for the properties these services expose through their standard describe APIs.
The one property that isn't covered is CloudTrail S3 delivery health. This requires a different API call than the standard trail description. DescribeTrails returns the configuration. GetTrailStatus returns the operational state. Most collectors call the first. Few call the second. The gap exists because the delivery status lives in a different API endpoint than the trail configuration.
The two compound gaps
Individual service controls fire independently. An attacker who disables GuardDuty triggers one finding. An attacker who also breaks Config delivery triggers another. An attacker who simultaneously breaks CloudTrail CloudWatch integration triggers a third. Each finding arrives separately. No compound finding says "multiple detection services are compromised simultaneously."
Detection response broken. GuardDuty findings are generated but not delivered (notification not configured). Config changes are recorded but not notified (SNS topic missing from delivery channel). CloudTrail events flow but don't reach CloudWatch for real-time detection (integration broken). Each service detects. None responds. A compound chain with threshold 2 fires when any two notification paths are broken. It catches the pattern that individual service checks miss. An attacker who breaks two notification paths simultaneously faces no alert escalation.
Detection quality decay. CloudTrail captures only writes (reads are invisible, an attacker's reconnaissance leaves no trail). Flow logs use default format (TCP flags and packet source addresses are missing. Forensic analysis can't determine connection direction or spoofing). GuardDuty has low detection method diversity. Cross-signal correlation fails because single-signal findings are less reliable. Retention is too short (evidence is gone before the investigation starts). Each is medium severity individually. Three of four simultaneously means the detection stack is structurally insufficient for post-incident forensics. Detection exists. Evidence doesn't.
The quality decay problem
Detection quality decay is subtler than detection absence. An absent detection service triggers an immediate finding. "GuardDuty is disabled" is a clear signal. A degraded detection service looks normal on every dashboard. The trail is enabled. The flow logs are active. GuardDuty is running. The dashboard is green.
But the trail only captures writes. The flow logs use the default format. GuardDuty has one detection method instead of three. Retention is seven days. Each setting is a choice someone made or a default someone didn't change. None triggers a critical finding individually. Together, they mean the organization's detection stack produces evidence that's incomplete in four dimensions simultaneously.
After an incident, the forensic team discovers: they can see what was modified but not what was read (write-only trail). They can see that traffic flowed but not the TCP flags or real source address (default flow log format). GuardDuty produced low-confidence single-signal findings that were dismissed (low method count). The detailed logs from the first three days of the intrusion aged out of the seven-day retention window before anyone started looking (short retention).
The detection stack was never disabled. It was never misconfigured in any way that triggers a critical finding. It was degraded across four dimensions to the point where it couldn't support the investigation it existed to enable.
The compound chain that detects this with threshold 3 out of 4 quality dimensions degraded fires when the detection stack crosses from "partially degraded" to "structurally insufficient." The threshold is deliberate: any 2 of 4 degraded is common and low-risk. Three of 4 means the detection stack is decorative.
The pattern
Every audit I run surfaces the same structural gap: individual controls check individual properties. Compound risks span properties across services. The individual checks pass. The compound path is unsafe.
Detection blindness follows the same pattern. Each detection service is independently configured. No service knows whether the other services are configured. An attacker who impairs defenses across multiple services faces multiple independent medium-severity findings. There is no one critical compound finding that says "your detection stack is structurally compromised."
The fix isn't more individual controls. It's compound chains that compose across services. The detection response chain asks: "are multiple notification paths broken simultaneously?" The quality decay chain asks: "are multiple evidence dimensions degraded simultaneously?" Neither question can be answered by checking one service at a time.
The 27-item detection blindness checklist, the CloudTrail delivery health control, and the two compound chains are implemented in Stave. The ghost reference pattern (configured but not functioning) appears across CloudTrail delivery, Secrets Manager rotation, SNS alarm actions, and CloudWatch subscription filters. The same archetype, different services, same structural shape.
Top comments (0)