DEV Community

Bala Paranj
Bala Paranj

Posted on

First Action After Compromise: Blind the Audit

#security #aws #cloud #appsec

✓ Human-authored analysis; AI used for formatting and proofreading.

If you're an attacker who's just landed in an AWS account, the most expensive thing about your future is detection. Every API call you make leaves a trail in CloudTrail, gets forwarded to a SIEM, becomes evidence the team uses to boot you out and reconstruct what you touched. The cost of your campaign rises linearly with the number of events on record.

So you make one call first:

aws cloudtrail stop-logging --name org-audit-trail
Enter fullscreen mode Exit fullscreen mode

The trail still exists. The console still shows it. aws cloudtrail describe-trails still returns it. The S3 bucket is still configured as the destination. To anyone glancing at the configuration, the audit infrastructure looks healthy.

Only aws cloudtrail get-trail-status reveals that IsLogging is false. New events stop landing in the bucket. The audit clock pauses. Whatever the attacker does next happens unobserved, and remains unobserved until somebody notices.

This pattern is established enough to have its own MITRE ATT&CK identifier: T1562.008 (Disable or Modify Cloud Logs). It appears in cloud incident write-ups across Mandiant's catalogue, AWS Security Bulletin guidance, and pen-test reports against organisations whose detection budget went into SIEM and not into CloudTrail-config monitoring.

A Stopped Trail

A stopped trail is structurally healthy. Every other configuration field is normal:

{
  "Name": "org-audit-trail",
  "S3BucketName": "acme-cloudtrail-archive",
  "IsMultiRegionTrail": true,
  "IsOrganizationTrail": true,
  "EventSelectors": [...],
  "InsightSelectors": [],
  "KmsKeyId": "arn:aws:kms:us-east-1:...",
  "LogFileValidationEnabled": true
}
Enter fullscreen mode Exit fullscreen mode

The only field that flips is the one get-trail-status returns separately:

{
  "IsLogging": false,
  "LatestDeliveryTime": "2026-01-01T00:00:00Z",
  "StartLoggingTime": "2025-09-15T00:00:00Z",
  "StopLoggingTime": "2026-01-01T00:00:00Z"
}
Enter fullscreen mode Exit fullscreen mode

Two timestamps and a boolean. That's the entire signal. Detection requires reading the right API — GetTrail and DescribeTrails both return the configuration but not the running state.

What the Attacker Buys

The audit blind window is the difference between a 30-day incident response and a 6-week one. Concrete things the attacker does inside that window:

  • Credential exfiltration. Read every secret in Secrets Manager, every parameter in SSM, every IAM user's credential report. The console actions show up in the SIEM immediately if you don't trust the attacker; the API calls show up in CloudTrail only if the trail is running.
  • Persistence creation. Create a new IAM user with AdministratorAccess, an access key for offline use, a Lambda function with a privileged execution role that the attacker invokes via API Gateway.
  • Evidence rewriting. Rotate KMS keys whose previous versions encrypted log files; the team can't decrypt the archive trail bucket even if the bucket itself survives.

None of these leave a CloudTrail event. All of them fail audit review because there is no audit. The investigator's first question: "what did this principal do between Jan 1 and Jan 8?" has no answer.

Why the Trail Looks Healthy

Operators who built the CloudTrail configuration usually checked their work. The bucket is encrypted, the trail is multi-region, log file validation is on, the bucket policy denies external principals. Every static configuration field passes audit review.

The mistake is treating "trail exists and is configured" as equivalent to "trail is running." They are not the same. The IAM permission that creates the trail (cloudtrail:CreateTrail) is different from the permission that toggles its state (cloudtrail:StopLogging, cloudtrail:StartLogging). Roles permitted to do one routinely have permission to do the other — incident-response runbooks frequently include "start logging if it stopped" without restricting the inverse.

The System Invariant

Every CloudTrail trail must have IsLogging: true,
and at least one trail in the account must be
multi-region.

In Stave's observation schema, the trail's running state is a first-class field on the asset:

{
  "id": "arn:aws:cloudtrail:us-east-1:111122223333:trail/org-audit-trail",
  "type": "aws_cloudtrail_trail",
  "properties": {
    "trail": {
      "name": "org-audit-trail",
      "is_logging": false,
      "is_multi_region_trail": true,
      "stopped_at": "2026-01-01T00:00:00Z",
      "stopped_by": "arn:aws:sts::111122223333:assumed-role/incident-actor/session"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

The collector that produces this observation calls GetTrailStatus for every trail, picks up the IsLogging boolean, and surfaces it directly. The diagnostic fields (stopped_at, stopped_by) carry forensic context the responder uses next; the predicate only reads the booleans.

The Stave Control 

id: CTL.CLOUDTRAIL.STOP.DETECT.001
name: CloudTrail Trails Must Be Actively Logging in All Regions
severity: critical
unsafe_predicate:
  any:
    - field: properties.trail.is_logging
      op: eq
      value: false
    - field: properties.trail.is_multi_region_trail
      op: eq
      value: false
Enter fullscreen mode Exit fullscreen mode

any — either the trail is stopped, or the trail is single-region (which leaves the other regions blind even when the configured one is running). Severity is critical because the trail being stopped is the configuration that determines whether every other finding in the account is investigable; the impact multiplies across the rest of the security posture.

Why Z3 Doesn't Help Here

This is a presence check, not a reachability question. The collector observes two booleans; the predicate is a two-leaf disjunction. There is no search space. There is no witness to enumerate.

Z3 earns its complexity when the question is "what specific request does this configuration admit?" — that's reachability over a permission set. The question here is "is the audit infrastructure running?" — a flat read on a status field.

Reproducing The Detection

A self-contained example is in the repo: stave/examples/cloudtrail-stop-logging/:

go run ./examples/cloudtrail-stop-logging before
Enter fullscreen mode Exit fullscreen mode

Captured stdout:

=== before (trail stopped) ===
  status: NON_COMPLIANT   total_assets=1   violations=1
  CTL.CLOUDTRAIL.STOP.DETECT.001 fired on 1 asset(s):
    - arn:aws:cloudtrail:us-east-1:111122223333:trail/org-audit-trail   severity=critical   exposure_score=100.00
  assertion: fires=true (expected) ✓
Enter fullscreen mode Exit fullscreen mode

After aws cloudtrail start-logging runs:

=== after  (trail re-started) ===
  status: COMPLIANT   total_assets=1   violations=0
  CTL.CLOUDTRAIL.STOP.DETECT.001: no findings
  assertion: fires=false (expected) ✓
Enter fullscreen mode Exit fullscreen mode

The before-fixture carries stopped_by — assumed-role/incident-actor/session — which is the forensic breadcrumb the responder uses next. CloudTrail itself recorded the StopLogging event before stopping; EventBridge can route that event to the SOC even after the trail goes silent, so the first unauthorised stop is detectable. The risk is detection latency. A six-hour latency is enough window for a determined attacker.

The Remediation

The remediation is a single command:

aws cloudtrail start-logging --name org-audit-trail
Enter fullscreen mode Exit fullscreen mode

A follow-up question every IR runbook should answer: what was logged in the audit-blind window? The answer is "nothing through this trail." Other data sources fill in the gap to varying degrees:

  • VPC Flow Logs record network connections; if the attacker called AWS APIs from inside the VPC, the flow records still exist.
  • Application logs in CloudWatch may capture attacker-side activity that shows up at the application layer (Lambda invocations, API Gateway requests).
  • GuardDuty runs on a separate data-plane and may have flagged anomalous behaviour during the window.
  • AWS Config records configuration changes asynchronously and survives the trail being stopped, because Config writes to its own S3 bucket with its own delivery channel.

None of these are a substitute for the trail. Together they shorten the audit-blind window's effective length.

The Prevention Lesson

Three layers, in order of leverage:

Service Control Policy denying StopLogging for any non-bootstrap principal. The strongest layer:

{
  "Sid": "DenyCloudTrailStopLogging",
  "Effect": "Deny",
  "Action": [
    "cloudtrail:StopLogging",
    "cloudtrail:DeleteTrail",
    "cloudtrail:UpdateTrail",
    "cloudtrail:PutEventSelectors"
  ],
  "Resource": "*",
  "Condition": {
    "ArnNotLike": {
      "aws:PrincipalArn": [
        "arn:aws:iam::*:role/CloudTrailAdministrator"
      ]
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

The deny applies to every principal except the explicit admin role. An attacker who lands on a compromised Lambda or compromised user has no path to StopLogging; the SCP rejects the call before it reaches CloudTrail's API.

EventBridge alarm on StopLogging events. The trail records the StopLogging event before going silent. Route that specific event to the SOC channel with a high-priority page:

{
  "source": ["aws.cloudtrail"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["cloudtrail.amazonaws.com"],
    "eventName": ["StopLogging", "DeleteTrail"]
  }
}
Enter fullscreen mode Exit fullscreen mode

The alarm catches the first unauthorised stop. Every moment the attacker waits is a moment the page rings.

stave apply in CI against the post-deploy observation snapshot. The example shipped with this article is the template. PRs that cause a trail's is_logging to go false produce exit code 3.

Checklist

  • [ ] Service Control Policy denies cloudtrail:StopLogging, cloudtrail:DeleteTrail, cloudtrail:UpdateTrail, cloudtrail:PutEventSelectors for any non-bootstrap principal
  • [ ] EventBridge rule matches StopLogging, DeleteTrail events from the management trail and pages the on-call SOC
  • [ ] At least one trail in the account is multi-region and management-event-enabled
  • [ ] stave apply runs in CI against post-deploy observations; PRs that introduce a stopped trail fail
  • [ ] CloudWatch metric filter on the trail's bucket tracks delivery cadence; gaps longer than 5 minutes page the on-call

The attacker's first action is StopLogging. The defender's first action should be making StopLogging hard to call without paging the on-call. The window between the call and the page is the budget the attacker has to spend; closing the window to seconds is how the rest of the security posture stays investigable.

Management Events Are Green. Object Events Don't Exist.

The control above asks one question: is the trail running? It's a presence check on is_logging and is_multi_region_trail. The CEL predicate is two booleans ANDed together. That's the headline-grabbing case where the attacker who calls StopLogging to evade detection.

The Bybit / Safe{WALLET} writeup (March 2025) makes a quieter point: "modifying objects in an S3 bucket can often be hard to detect because, by default, CloudTrail logging for S3 object activity is disabled." That is a different gap. Management events log CreateBucket, PutBucketPolicy, DeleteBucket. Data events log GetObject, PutObject, DeleteObject. Most production trails enable management events and stop there — data events are high-volume, expensive, and opt-in.

The trail in data-events-before/ is a normal production CloudTrail:

trail_name: org-management-trail
is_logging: true
is_multi_region_trail: true
include_global_service_events: true
log_file_validation_enabled: true
event_selectors:
  - read_write_type: All
    include_management_events: true
    data_resources: []         # ← empty
Enter fullscreen mode Exit fullscreen mode

Five booleans say "logging is healthy." The compliance dashboard shows green. The CEL control above (CTL.CLOUDTRAIL.STOP.DETECT.001) reports no findings — because, by its definition, there is nothing wrong: is_logging=true and is_multi_region_trail=true. The check passes.

But the account contains three production buckets — company-frontend-prod (serves JavaScript via CloudFront), customer-data-prod (PII), and financial-records (financial data) — and none of them appear in any trail's data_resources. An attacker who modifies app.js in the frontend bucket leaves no CloudTrail record. The security team's dashboard stays green.

The case for Z3 on this question

The earlier section "Why Z3 Doesn't Help Here" is honest about the original control: presence checks have no search space, no witnesses to enumerate. CEL is the right shape.

The data-events question is different. It asks: is there a sensitive bucket whose object operations are not covered by any trail's data_resources? That has the shape Z3 is built for — quantifier over buckets, predicate combining tag classification and trail coverage. The prover at examples/cloudtrail-stop-logging/z3prove/ runs two queries:

Query 1 — coverage gap. ∃ bucket b: b is sensitive ∧ no trail covers b.

=== data-events-before (mgmt logging on, data events off) ===
  trails observed:  1
    - org-management-trail   is_logging=true   data_resource_patterns=0
  buckets observed: 4
    - company-frontend-prod   environment=production   data_classification=public   sensitive=true
    - customer-data-prod      environment=production   data_classification=pii      sensitive=true
    - financial-records       environment=production   data_classification=financial sensitive=true
    - company-frontend-dev    environment=development  data_classification=          sensitive=false

  --- S3 Data Event Logging Gap ---
  verdict: SAT
  witness: arn:aws:s3:::company-frontend-prod
           (sensitive=true, classification=public, environment=production,
            data_event_coverage=NONE)
  rationale: 3 of 3 sensitive buckets fall outside every trail's data_resources
Enter fullscreen mode Exit fullscreen mode

Three of three sensitive buckets are uncovered. Z3 picks one as the witness. The prover doesn't care which; any unconvered sensitive bucket discharges the existential.

Query 2 — compound: write access without audit. This is the conjunction that makes the gap matter: write_admitted ∧ ¬covered. The write_admitted half is discharged against an IAM policy with a prefix wildcard; the ¬covered half is discharged against the trail's event selectors. The compound is the supply chain attack path Bybit took: modify production JavaScript, leave no CloudTrail record, $1.5B redirected.

The fix is targeted, not comprehensive

Don't enable data events on every bucket. That's cost-prohibitive and probably useless. Enable data events on the buckets that contain PII, serve production frontends, or hold financial records. The remediated fixture is a one-statement change:

event_selectors:
  - read_write_type: All
    include_management_events: true
    data_resources:
      - type: AWS::S3::Object
        values:
          - arn:aws:s3:::company-frontend-prod/
          - arn:aws:s3:::customer-data-prod/
          - arn:aws:s3:::financial-records/
Enter fullscreen mode Exit fullscreen mode

Three ARN prefixes. Z3 reports UNSAT on both queries:

=== data-events-after  (data events scoped to sensitive buckets) ===
  --- S3 Data Event Logging Gap ---
  verdict: UNSAT
  rationale: every sensitive bucket is covered by at least one trail's data_resources

  --- Compound: Write Access + No Audit Trail ---
  verdict: UNSAT
  rationale: no logging gap → compound trivially UNSAT
Enter fullscreen mode Exit fullscreen mode

The teaching beat: management events answer "is logging running?" Data events answer "is logging covering the buckets that matter?" The first question is one boolean. The second is a quantifier over a sensitivity-tagged bucket inventory and a coverage relation. Different question, different prover.

The CEL control above is correct as written. It catches the attacker who runs StopLogging. The data-event question is the one a heuristic scanner can't ask — it requires reasoning over which sensitive buckets fall outside which trail's data_resources patterns, across the join of two asset types. Z3 closes that gap.

The example at stave/examples/cloudtrail-stop-logging/ is a self-contained Go program that loads two fixture snapshots, runs pkg/stave.Apply, asserts that CTL.CLOUDTRAIL.STOP.DETECT.001 fires on the trail-stopped fixture and is silent on the running one, and exits zero when both assertions hold. Stave detects this pattern and 31 other H1-grounded scenarios from local AWS configuration snapshots, with no cloud credentials.

Top comments (0)