AdminPackStudio

Posted on Oct 7

Conditional Access baseline for Entra ID: naming, pilot groups, report-only, Insights, break-glass and rollout waves

#microsoft #azure #security #sysadmin

Conditional Access baseline for Entra ID: from report-only to enforce, step by step

Conditional Access (CA) is the most powerful switch in a Microsoft 365 tenant, and the easiest one to get wrong. One policy scoped to All users and All cloud apps, set to On on a Friday afternoon, and Monday starts with a helpdesk queue full of people who can't open Outlook. Sometimes that includes the admins.

The policies themselves are rarely the issue. The trouble usually comes from the rollout: no naming standard, no pilot group, report-only skipped or never actually reviewed, no emergency access, and a big-bang switch to "everyone."

This guide is the long version of a baseline that avoids that. It covers:

What a CA baseline is (and isn't)
Prerequisites: licenses, roles, inventory
A naming convention you'll still like in a year
Pilot groups that actually tell you something
Report-only mode, done properly
What If and Insights and reporting
Break-glass (emergency access) accounts
The starter policy set
Expanding in waves
Troubleshooting table
Ongoing operating rhythm

If you only want the five-minute version, the short post is here: Conditional Access without the Monday lockout: report-only → enforce.

What a Conditional Access baseline is (and isn't)

A CA baseline is a small set of policies that every tenant should have, plus the process you use to change them safely. The policies are the easy part. The process is what keeps you from locking people out.

What CA does: it evaluates a sign-in to a cloud app (who, what app, what device, where from, how risky) and decides allow, require something (MFA, a compliant device, an authentication strength), block, or limit the session.

What CA does not do:

It doesn't configure devices. Intune does that. CA only reads the device's compliance state as a signal.
It doesn't fix a broken MFA registration process. It will expose one, loudly.
It doesn't apply to things that never touch Entra sign-in (an on-prem file share over SMB, for example).

That last distinction matters for planning. If you plan to require a compliant device, your Intune compliance has to be green first. Otherwise CA just turns every compliance gap into an access outage.

Prerequisites: licenses, roles, and an inventory

Before you design anything, check these three things.

Licenses

Conditional Access needs Microsoft Entra ID P1 for every user the policies cover. P1 is included in Microsoft 365 Business Premium and E3/E5 suites. Risk-based conditions (sign-in risk, user risk) need P2. Build the baseline on P1 and treat risk policies as a later upgrade. Don't hold the baseline back waiting for them.

If your tenant runs on security defaults, know that you'll have to turn them off to use CA. Plan your replacement MFA policy before you do, so there's no gap where nothing enforces MFA.

Roles

Day-to-day CA work doesn't need Global Administrator:

Task	Role
Create and edit policies	Conditional Access Administrator
Read policies, logs, exports	Security Reader or Global Reader
Emergency recovery	Global Administrator (break-glass only)

Inventory what already exists

Many tenants already have a few CA policies from a previous admin, a consultant, or Microsoft-managed policies. Before you add anything, write down every existing policy: its state (On / Report-only / Off), who it targets, and what it grants. Export it to CSV so you have a "before" picture for change control. Any On policy that nobody can explain should get reviewed before you add new ones on top of it.

A Conditional Access naming convention that scales

At 3 a.m. during an incident, you'll be scanning a list of 25 policies to find the one blocking finance. Names like MFA, Test, and New policy (2) are no help then.

Use a structured name that reads like a change ticket:

CA - <Who> - <Control> - <Apps/Scope> - <Phase> - v<n>

Example	Reads as
`CA - AllUsers - Require MFA - O365 - ReportOnly - v1`	Broad MFA design, still observing
`CA - Admins - Require MFA - AllApps - Enforce - v2`	Admin hardening, live, second revision
`CA - AllUsers - Block Legacy Auth - AllApps - Enforce - v1`	Legacy protocols blocked
`CA - Guests - Require MFA - AllApps - Pilot - v1`	Guest policy in pilot
`CA - Windows - Require Compliant - O365 - Wave2 - v1`	Device gate, second rollout wave

A few rules that make the convention stick:

Same field order every time. Alphabetical sorting then groups policies by audience automatically.
Pick one source of truth for phase. Either put the phase in the name and rename at each step, or keep names stable and rely on the portal's state column plus your export. Don't mix the two habits.
Use the Description field for owner, ticket number, last review date and a warning: Owner: SecOps | Ticket: CHG-1042 | Reviewed: 2026-10-06 | Do not widen scope without pilot sign-off.
Bump the version whenever the grant or the scope changes. That makes before-and-after exports easy to diff.

Name your groups the same way: CA-Pilot-Wave0, CA-Wave2-Finance, CA-Excl-LegacyApp-Exp2026-11-30. An exclusion group with its expiry date in the name is much harder to forget.

Pilot groups: who goes first and why

A pilot only helps if it's realistic. If the pilot is three IT admins on brand-new laptops, every policy will look fine until it reaches the warehouse.

A good pilot group (CA-Pilot-Wave0) has:

5–15 people across roles: IT, someone in finance, someone who travels or works in the field, an executive assistant (they act on behalf of others and hit edge cases early).
A mix of devices: managed Windows, a Mac if you have them, a phone using Outlook mobile, and if possible one device you know is noncompliant, so you can see what failure looks like.
People who know they're in a pilot. Send a short note: what's changing, when, and who to call.

Scope rules for the pilot:

Include the pilot group only. Never start with "All users" on a Block or a hard device requirement.
Exclude break-glass accounts from day one, even in pilot (see below).
For harsh grants (compliant device, block), start with a specific app such as Office 365 instead of "All cloud apps." Widen the app scope later, as a separate change.

Report-only mode: how to use it properly

Report-only is the most useful CA feature and the most often wasted. Admins turn it on, never look at the results, and then flip to On a week later. All that tells you is that a week went by.

What report-only does: the policy is evaluated on every matching sign-in and the result is logged, but nothing is enforced. Each sign-in shows outcomes like Report-only: Success, Report-only: Failure, or Report-only: User action required (for example, the user would have been asked for MFA).

How to run a report-only window:

Create the policy with Enable policy = Report-only. Don't create it as On "just to test."
Leave it running for at least 3–5 business days. You want a Monday in there, plus month-end if the policy touches finance apps.
Review the results daily (next section shows where).
Classify every report-only failure as either:
- Expected: this person genuinely isn't ready (no MFA method, noncompliant device). That's a remediation ticket, not a reason to weaken the policy.
- Bug: the policy hits something it shouldn't (a service account, a guest, an app you didn't mean to include, an Intune enrollment flow). That's an assignment fix.
Set an exit bar before you start. For example: "≥95% of pilot users have a registered MFA method, and zero unexplained report-only failures for 3 consecutive days."

A word of caution: a policy that requires a compliant device can trigger device-related prompts during report-only on some platforms. Read the current Microsoft documentation for any device-based policy before you put it in report-only, and test it on a pilot device first.

What If and Insights and reporting: reading the results

Two tools cover most of the investigation work. Use both.

What If (before and after every change)

Entra admin center → Protection → Conditional Access → Policies → What If.

Run these scenarios on every new or changed policy:

Scenario	Expected result
Pilot user + Office 365 + normal location	Your policy applies
Break-glass account + any app	Your policy does not apply
Non-pilot user	Your policy does not apply (while still pilot-scoped)
Guest user (if guests are in scope)	Matches your guest design

If What If surprises you, fix the assignment before you go any further.

Sign-in logs (single-user truth)

Entra → Monitoring & health → Sign-in logs. Open a sign-in and look at the Conditional Access and Report-only tabs. You'll see every policy that was evaluated, whether it applied, and which grant control passed or failed. This is where you confirm a specific user's story ("I got blocked at 8:05").

Insights and reporting (the aggregate view)

Entra → Protection → Conditional Access → Insights and reporting shows the combined effect of one or more policies across all sign-ins over a time range, broken down by user, app, and result.

Practical notes:

The workbook reads sign-in logs from a Log Analytics workspace. If you haven't set up diagnostic settings to send sign-in logs there, set that up first, or the view will be empty.
Filter to one policy at a time during a report-only window.
Sort by Failure and User action required, then open 3–5 sample sign-ins for each pattern. Don't try to read every row.
Screenshot or export the summary for your change ticket. "Report-only reviewed, 2 false failures fixed, 0 outstanding" is the evidence your CAB wants.

If Insights shows a pile of "would block" results for normal, expected work, you aren't ready to enforce, no matter how long the policy has been in report-only.

Break-glass accounts: before you enforce anything

A break-glass (emergency access) account is how you get back in when CA, MFA, or a bad change locks out every admin. Without one, your rollback plan depends on another Global Admin being reachable and not locked out too.

Baseline setup:

At least two cloud-only accounts on the *.onmicrosoft.com domain (not synced from on-prem AD, so an AD or sync problem can't take them out).
Global Administrator, permanently assigned. Not PIM-eligible, because activation may depend on the same systems that are broken.
Excluded from your CA policies, ideally through one CA-Excl-BreakGlass group that you add to every policy's exclusions.
A strong, phishing-resistant sign-in method. Microsoft now requires MFA to sign in to its admin portals, and that applies to emergency accounts as well, so plan for something like a FIDO2 security key stored with the sealed credentials. Check current Microsoft guidance on emergency access accounts when you set this up.
Credentials kept offline in a sealed, documented process, so that two people can retrieve them without depending on the tenant being up.
Monitored. Every break-glass sign-in should raise an alert and be treated as an incident until explained.
Tested quarterly: sign in, open the CA policy list, sign out, write down that the test happened.

The emergency disable procedure (print it and keep it with the credentials):

Sign in as break-glass from a known clean device.
Go to Conditional Access → Policies → the offending policy.
Set it to Report-only (keeps logging for root cause) or Off.
Post in the incident channel: what changed, when, who.
After recovery: root cause, rotate the break-glass credential, and reintroduce the fix through the normal report-only path.

Exclusion hygiene: break-glass is the only permanent exclusion that's acceptable. Every other exclusion (a legacy app, a user stuck mid-migration) should be a group with an owner, a ticket and an expiry date in its description. Exclusions that never get removed are how a CA baseline slowly stops protecting anything.

The starter baseline policy set

Don't enable everything on day one. Most tenants get the most value from these, introduced one at a time, each going through pilot → report-only → enforce:

#	Policy	Why	Watch for
1	Require MFA for admin roles	Highest-value accounts first; small blast radius	Target directory roles, exclude break-glass
2	Block legacy authentication	Legacy protocols can't do MFA, so they bypass it	Old Outlook clients, IMAP/POP, scan-to-email devices
3	Require MFA for all users	The core control	MFA registration gaps; use Temporary Access Pass for stragglers
4	Require MFA for guests	B2B accounts are often forgotten	Separate policy, separate pilot
5	Require compliant device (Windows first)	Only managed, healthy devices reach data	Intune compliance must be green first

Policy 5 is the one most likely to cause a Monday lockout, because it depends on a whole separate system (Intune) being in good shape. Two things to settle before it goes beyond pilot:

Compliance has to be reliably green. Devices that are still encrypting, haven't checked in, or are inside a compliance grace period are common causes of false "not compliant" blocks. I covered one of the most common in BitLocker still encrypting: why Intune marks you noncompliant (and how grace periods save Monday).
Don't create a circular dependency with enrollment. If a "require compliant device" policy covers the apps a device needs in order to enroll (Microsoft Intune / Microsoft Intune Enrollment), new devices can't get compliant because they can't finish enrolling. Check Microsoft's current guidance on which enrollment apps to exclude.

(If your enrollment and compliance side needs work first, there's a separate Intune & M365 Admin Starter Pack for that, linked at the end.)

Rolling out in waves instead of a big bang

Once a policy has been enforced on the pilot for a couple of days without drama, expand it in waves. Use nested groups: add wave groups into the policy's include list instead of switching to "All users" and adding exclusions.

Wave	Who	Enter when	Hold for
0	Pilot (`CA-Pilot-Wave0`)	Report-only clean, What If verified	48 hours enforced
1	IT + Security	Wave 0 had no emergency rollback	3 business days
2	One department (pick a cooperative one)	Helpdesk ticket rate normal	1 week
3	Remaining staff	Remediation backlog cleared	Monitor 2 weeks
4	Guests / vendors	Separate guest policy validated	Ongoing

Rules for each wave:

Tell people before the wave, not after. One paragraph: what changes, what they'll see, what to do if they're blocked.
Watch the helpdesk queue and Insights for 48 hours after each wave.
Have a pre-agreed rollback trigger, for example "more than 5 lockout tickets in an hour from the new wave → remove that wave group, investigate." Removing one wave group is a much smaller, safer change than turning the whole policy off.
Don't expand on Fridays or right before holidays or month-end close.

When the final wave is in, you can switch the include list to "All users" (still excluding break-glass) and retire the wave groups, or keep the waves for the next policy. Either works, as long as it's written down.

Conditional Access troubleshooting table

These are the tickets you'll see during and after rollout. Rule zero: don't disable CA org-wide as a first response. Scope the fix to the affected user or wave and time-box it.

Triage in five minutes: get the error screenshot with the Correlation ID / Request ID → open that user's sign-in log → Conditional Access tab → find which policy shows Failure → reproduce in What If → then fix.

Symptom	Likely cause	First check	Safe fix
"More information required" / MFA registration loop	No usable MFA method, or registration blocked by policy	User's authentication methods; auth methods policy	Issue a Temporary Access Pass; finish registration on a trusted network
"You can't get there from here" / device not compliant	Device not enrolled, not synced, or compliance not yet evaluated	Intune device record: compliance state + last check-in	Sync the device, fix the failing setting; time-boxed exclusion group (≤48h) with ticket if business-critical
Compliant in Intune but CA still blocks	Device identity mismatch (hybrid join vs Entra join), or browser not passing device info	Device ID in sign-in log vs Intune; browser used	Use a supported browser / sign-in method that passes device identity; fix join state
Old Outlook / IMAP / POP / SMTP app fails, new Outlook works	Legacy authentication blocked	Sign-in log Client app column	Move to a modern-auth client; temporary exception group with expiry for true business blockers
Scan-to-email printer or app stops sending	Device uses basic SMTP auth	Sign-in log for the service account	Use a supported relay/connector method; don't permanently exclude the account
Guest can't open Teams / SharePoint	Guest policy requires MFA they haven't set up, or cross-tenant settings	Guest's sign-in log; cross-tenant access settings	Guest completes MFA registration; adjust cross-tenant trust deliberately
New devices stuck during Autopilot / enrollment	Compliant-device policy covers enrollment apps (circular dependency)	Which policy fails on Microsoft Intune Enrollment sign-ins	Exclude enrollment apps from the device policy per Microsoft guidance
Traveling user blocked or prompted unexpectedly	Named location / country rule; VPN egress IP changed	Sign-in log location + IP; named location definitions	Update named locations; time-boxed traveler exception
Named admin locked out of portal	Admin policy too strict, MFA method broken	Which policy failed; admin's auth methods	Second admin sets policy to Report-only or adds timed exclusion; break-glass only if no other admin can act
Report-only shows failures nobody can explain	Policy scope wider than intended (apps, users, platforms)	What If with a sample user	Fix assignment before enforcing; that's what report-only is for

Severity guide:

Severity	Example	Response
SEV1	All admins locked out	Break-glass → offending policy to Report-only → incident bridge
SEV2	A whole department can't reach email	Remove that wave group or scope a ≤4h exclusion; fix the root cause
SEV3	One user, one app	Standard ticket; no tenant-wide changes
SEV4	Report-only noise	Tune during business hours

Keeping the baseline healthy

A baseline isn't a one-time project. A light routine keeps it from drifting:

Cadence	Action
Weekly	Skim CA failures in Insights; check for new report-only failures on anything in flight
Monthly	Export all policies to CSV and diff against last month; review exclusion groups for expired entries
Quarterly	Test a break-glass sign-in; run a tabletop of the emergency disable procedure
After any big Intune or identity change	Put affected device-based policies back through a short report-only check

Checklist before any policy goes to On:

[ ] Name and Description follow the convention (owner, ticket, review date)
[ ] Include = pilot or wave group, not All users
[ ] Exclude = break-glass group (+ documented, expiring exceptions only)
[ ] App scope isn't accidentally "All cloud apps" with a Block grant
[ ] Report-only ran ≥3 business days and Insights was reviewed
[ ] What If verified for pilot user, break-glass and non-pilot user
[ ] Remediation tickets for false failures are closed
[ ] Rollback trigger and owner written in the change ticket

DEV Community