Most of the cloud security incidents we get called into have nothing to do with a novel exploit. Someone opened a security group to 0.0.0.0/0 for a vendor's remote session and never closed it. A storage account got flipped to public so a contractor could pull a file. An admin was excluded from conditional access "temporarily" in March. A mailbox forwarding rule was added and nobody noticed because nobody was looking.
None of that shows up in a vulnerability scan. It shows up in configuration, and configuration changes constantly, in three or four different consoles, made by people who had a good reason at the time. This post is part of our Practical AI in Cyber Security series, and it covers the service we run for clients: a scheduled job that reads the current state of AWS, Azure and Microsoft 365, compares it to a stored baseline, and emails a human when something has moved.
Why point-in-time auditing stopped working
The traditional pattern is an annual review where someone screenshots portal settings for the auditor. Cloud-first teams have been moving away from that towards continuous, code-driven checks that run against AWS and Azure on a schedule, so the audit is a job that runs rather than a scramble in the weeks before certification (r/Information_Security discussion, Aug 2026).
The control pattern that thread describes is worth stealing directly: state the requirement, tag the resource, run the check, report the result. It is boring and it works across every control domain we care about. Each of our checks is written that way, as a small independent unit with a stated requirement and a pass or fail, rather than as a giant "security score" that nobody can act on.
There is a second reason to build this yourself rather than lean entirely on the native tools. Most Australian organisations we work with are not on one platform. They have Microsoft 365 for identity and email, some Azure, and AWS from an acquisition or a dev team that got there first, and the relative growth of the big three suggests that mix is not resolving itself any time soon (r/Mag7shares, Aug 2026). Three consoles with three different opinions about what "secure" means is how things get missed. One consolidated report, one owner, one weekly read.
What we actually check
The check list is deliberately narrow. These are the things that drift, that matter, and that a person can fix in an afternoon.
- Identity in AWS and Azure: IAM users with console access and no MFA, long-lived access keys, keys that have not rotated, roles with wildcard permissions, orphaned service principals.
- Network exposure: security groups and NSGs allowing inbound from anywhere, especially on 22, 3389, 1433 and 3306. Public IPs on resources that had none last week.
- Storage: S3 buckets and Azure blob containers with public access, anonymous read enabled, or encryption settings changed.
- Microsoft 365 identity: conditional access policies added, disabled or scoped down, break-glass exclusions, MFA registration state versus MFA enforcement (they are not the same thing and the gap is where people live), legacy authentication re-enabled.
- Exchange Online: mailbox forwarding rules to external domains, new transport rules, mailbox delegation changes.
- Entra ID app surface: new enterprise app registrations, OAuth consent grants and the scopes they were granted, changes to admin role membership including eligible assignments in PIM.
- Cost and orphan drift: resources that stopped being used but did not stop being billed. One operator running an identical 24/7 workload found AWS came in around 30% more expensive than the alternatives (r/automation, Sep 2026), which is a fair reminder that an unnoticed left-running instance is a real cost, not just an untidy inventory.
That last one is not strictly a security check, but it uses the same snapshot and it is the item that most often gets a client to actually read the email.
The architecture
The design is simple enough to describe in a paragraph, which is part of the point.
- A scheduled worker (we run these hourly for identity checks, daily for everything else) authenticates to each tenant using a read-only role: an AWS IAM role assumed cross-account, an Azure reader plus a Graph app registration with directory and policy read scopes. No write permissions anywhere.
- Each collector pulls the relevant state and normalises it into a flat JSON document per platform. Resource identifier, control name, observed value, timestamp.
- That document is diffed against the last stored snapshot. Any difference is a drift event with a before value and an after value.
- Drift events go to a model that writes the summary a human reads: what changed, who changed it if the audit log says, and why it matters for this control.
- The report goes to email and to a Teams channel. Nothing is remediated automatically.
Where clients already have infrastructure as code, the diff is even cleaner, because the declared template is the baseline and live state is the thing that has wandered. Practitioners in that same ISO 27001 thread make the point about Bicep specifically. Full IaC coverage is rare in the environments we inherit, so snapshots plus drift detection give you the same signal without requiring a rebuild first.
Where the AI part earns its keep
The detection is deterministic. Rules, diffs, comparisons. We do not use a model to decide whether something is a finding, because a model that occasionally misses a public bucket is worse than no check at all.
The model writes the explanation. Raw drift output is unreadable: a JSON blob showing that conditionalAccessPolicy/7f3a... moved from enabled to enabledForReportingButNotEnforced. The summary the client gets says that the policy requiring MFA for admin sign-ins from outside Australia was switched to report-only mode on Tuesday afternoon, that it currently blocks nothing, and that the change was made by a named admin account. Same fact, but one of them gets actioned.
The model also groups related drift. Twelve NSG changes from a single deployment become one paragraph rather than twelve alerts, which is the difference between a report people read and a filter rule people write.
We keep the model on a short leash. It gets the drift record and the control description, and it writes prose from those. It does not query the tenant, it does not decide severity on its own, and every summary sits next to the raw before and after values so the reader can check it. When the model is wrong, it is wrong about wording, not about whether something changed.
Detect and notify, not auto-fix
We deliberately do not remediate automatically. Experienced operators are sceptical of automated remediation for good reason, as one put it in that thread: "I'm personally not a fan of automated remediations as it's hard to do."
Our experience matches. A rule that automatically closes an open security group will eventually close the one that a legitimate integration depends on, at 2am, in the environment where nobody documented the dependency. In tenants with clinical or health administration workloads, that risk is not acceptable at all. Any change to an operational system needs a person to sign it off before it happens. That is a design requirement for us, not a footnote.
So the output is a decision point, not an action: here is what changed, here is what we think it means, here is the button that says "expected, add to baseline" or "not expected, raise a ticket".
The first run rule: never silently seed
This one is worth stating on its own because it is the mistake we see most often in home-grown versions.
The first time the service runs against a tenant, there is no prior snapshot, so nothing is technically "drift". The tempting behaviour is to quietly store that state as the baseline and start alerting from run two. That means every pre-existing misconfiguration, every public bucket and every stale global admin becomes invisible by definition, permanently blessed as normal.
Our first run emails the complete baseline instead. Every public storage container, every account without MFA, every open security group, every external forwarding rule that exists on day one. It is usually an uncomfortable document, it is often the most valuable single report a client gets from us, and it forces an explicit decision on each item before anything becomes "normal".
Costs and limitations
Being honest about what this does not do:
It only sees what the APIs expose. Graph rate limits mean a large tenant's full Microsoft 365 sweep takes longer than you would like, and some settings are only readable through interfaces that change without notice. Collectors break, so we monitor the collectors themselves and treat a silent run as a failure.
It is noisy in environments with active deployment pipelines until the rules are tuned. Expect two to four weeks of tuning before the signal to noise ratio is good, and expect to keep tuning as the environment changes.
New rules should never fire their first alert against production. Local emulators for AWS, Azure and GCP let you develop and regression-test drift rules offline (flowg.cloud, Aug 2026), which we use for the AWS side in particular. The same write-up is candid that emulators do not validate real authentication, so the last test always happens against a real tenant, in read-only mode, before a rule goes live.
Cost-wise, the compute is trivial. A scheduled worker plus snapshot storage runs at a few dollars a month. The model calls for summarisation are similarly small because they only fire when there is drift. The real cost is the human time in the first month, building the check set for your environment and clearing the day-one baseline. Budget for that honestly, because a posture service nobody triages is just an expensive mail rule.
PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.
Originally published at picnet.com.au.
Top comments (0)