From Cloud Audit Logs to Real-Time Threat Detection with BigQuery and Chronicle SIEM

#googlecloud #cloudsecurity #cybersecurity #security

Introduction

Cloud Audit Logs help you track every cloud action. They show who did what, when, and where it happened. These logs are your foundation for strong visibility and control.

However, storing logs alone isn’t enough for compliance. Security teams need to analyze patterns, detect threats, and respond fast. That’s where smarter, automated tools become essential.

In this article, you’ll learn to build a complete pipeline. It combines Cloud Audit Logs, BigQuery, and Chronicle SIEM. This setup helps you ingest, normalize, and query logs at scale.

You’ll also learn how to detect risky activity using patterns. Then, you’ll visualize anomalies and enrich results with identity data. The goal is simple—turn raw logs into real-time security insight.

Why Go Beyond Logging?

This table shows why basic logging is not enough. It highlights common cloud audit challenges and their ideal solutions.

Each challenge reflects a gap in visibility or action. The tools listed help you filter, correlate, and automate responses. This helps security teams move from raw data to real insights.

Challenge	Solution
Too many logs, not enough signal	UDM normalization + SQL filters in BigQuery
Manual investigation workflows	Chronicle timeline + ML insights
Compliance without visibility	Structured queries, dashboards, audit pipelines
Siloed audit logs	Centralized correlation across services

Reference Architecture

This reference architecture shows the data flow clearly. It starts with GCP Audit Logs capturing cloud events. Logs move through Cloud Logging and Pub/Sub for routing. Then, BigQuery handles large-scale analytics and queries.

Chronicle SIEM performs real-time threat detection and context. This setup blends batch analytics with instant event correlation. It helps you cover compliance and threat response effectively.

Step-by-Step: Building the Pipeline

This section guides you through creating a complete logging and detection pipeline. You’ll learn each step to collect, analyze, and respond to security events efficiently.

1. Enable Cloud Audit Logs for All Resources

Start by enabling Cloud Audit Logs across all resources. Make sure both Admin Activity and Data Access logs are turned on. Admin logs show who changed settings or roles. Data Access logs track who viewed or read data. Enabling both gives full visibility into your cloud actions.

gcloud logging sinks create audit-log-sink \
bigquery.googleapis.com/projects/myproject/datasets/audit_dataset --logfilter='logName:"cloudaudit.googleapis.com"'

This includes IAM, GKE, BigQuery, Storage, Compute Engine, Cloud Run, etc.

2. Normalize Logs in BigQuery for Querying

Once logs are enabled, you’ll need to make them useful. BigQuery lets you clean and structure logs for faster searches. You can pull out key fields like user, action, and resource. This helps you run clear queries and build reports easily.

Example: Flatten IAM permission changes

SELECT
protopayload_auditlog.authenticationInfo.principalEmail AS actor,
protopayload_auditlog.resourceName AS resource,
protopayload_auditlog.methodName AS action,
TIMESTAMP_SECONDS(receiveTimestamp.seconds) AS ts
FROM
`audit_dataset.cloudaudit_googleapis_com_activity`
WHERE
  protopayload_auditlog.methodName CONTAINS "SetIamPolicy"
  AND resource CONTAINS "bigquery"

Use scheduled queries or views for dashboard generation in Looker Studio.

3. Ingest Logs into Chronicle for Real-Time Enrichment

Next, send your logs to Chronicle for deeper analysis. Use Pub/Sub or the Chronicle API to forward logs in real time. Chronicle adds context, links events, and highlights risky patterns. This step turns raw logs into useful, security-ready data.

For Example:

gcloud pubsub topics create chronicle-audit
gcloud logging sinks create chronicle-sink \
pubsub.googleapis.com/projects/my-project/topics/chronicle-audit

Format logs using Unified Data Model (UDM) before sending to Chronicle:

{
"metadata": {
 "productname": "GCP",
 "event_type": "IAM_POLICY_CHANGE"
},
"principal": {
  "user": { "userid": "admin@example.com" }
},
"target": {
 "resource": "bigquery/project-id/dataset"
 }
}

Send logs to Chronicle UDM API:

curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d @log.json \
  https://backstory.googleapis.com/v1/udmevents

4. Write Chronicle Rules to Detect Abuse

Write custom rules in Chronicle using YARA-L syntax. This helps detect abuse, risky changes, or rare behaviour

rule ExcessivePolicyChanges
{
  meta:
    author = "advait@gcpsec"
    description = "Detects multiple IAM policy changes in short time"

  events:
    $e.metadata.event_type == "IAM_POLICY_CHANGE"
    and $e.metadata.product_name == "GCP"
    and count($e) > 5 over 10m
}

Chronicle alerts the SOC if abuse patterns are detected.

5. Build Contextual Timeline Investigations

Use Chronicle’s timeline to trace user activity over time. It helps connect events and understand how incidents unfold.

Lateral movement
Privilege escalation
Suspicious API access (For example, getIamPolicy, testIamPermissions)
Admin activity outside business hours

Example query:

FETCH events
  WHERE principal.user.userid = "admin@example.com"
  AND metadata.event_timestamp BETWEEN "2025-06-18T00:00:00Z" AND "2025-06-18T06:00:00Z"
RETURN action, resource, src_ip

Use Case: Detecting Insider Threat in BigQuery

An admin uses SetIamPolicy to grant full BigQuery access.
Shortly after, they access sensitive datasets during off-hours.
These actions happen outside normal user behavior patterns.
Chronicle links the role change and unusual data access together.
Identity data adds context like user role and device location.
An alert is raised based on pattern, time, and privilege use.

This type of correlation isn’t visible in raw logs alone.

Advanced Audit Log Analytics Checklist

This checklist shows how to turn logs into insight. Each task pairs with a tool built for speed and scale. You’ll cover everything from enabling logs to threat detection and reporting. It helps teams stay compliant, alert, and audit-ready.

Task	Tool
Enable audit logs for all resources	Cloud Logging
Ingest and normalize logs	BigQuery + UDM
Real-time alerting and detection	Chronicle SIEM + YARA-L
Timeline-based investigations	Chronicle Timeline + Entity Context
Long-term compliance reporting	BigQuery + Looker Studio
Threat hunting + enrichment	Chronicle UDM Search