Rolling out a Conditional Access change without locking anyone out - report-only, What If and sign-in log checklist

#Why a checklist

Conditional Access failures are rarely syntax errors. They come from evaluation you didn't predict: a device-compliance grant that hits unmanaged build agents, a location condition that misses an egress IP, a "All resources" policy that catches a legacy client's token refresh. Report-only mode shows you that before enforcement, but only if you read the results properly. This checklist is the sequence we use for any change that adds a block, a new grant control or widens scope.

#1. Before you touch anything

• Snapshot every policy to source control. You need the JSON to diff and to restore.

az rest --method get \
  --url "https://graph.microsoft.com/v1.0/identity/conditionalAccess/policies" \
  > ca-policies-$(date +%F).json

jq -r '.value[] | [.state, .displayName, .id] | @tsv' ca-policies-$(date +%F).json

Connect-MgGraph -Scopes 'Policy.Read.All'
Get-MgIdentityConditionalAccessPolicy -All |
  Select-Object State, DisplayName, Id | Sort-Object DisplayName

• Confirm break-glass accounts are excluded from the new policy and from every existing block policy, that their credentials work, and that sign-ins from them alert.
• List named locations and verify the egress IPs are current. Stale ranges are the most common cause of "everyone in office X is blocked".

az rest --method get \
  --url "https://graph.microsoft.com/v1.0/identity/conditionalAccess/namedLocations" \
  --query "value[].{name:displayName, type:'@odata.type'}" -o table

• Inventory non-interactive and workload sign-ins that the policy could match. Service accounts, sync accounts, kiosk devices and legacy apps live here.

AADNonInteractiveUserSignInLogs
| where TimeGenerated > ago(14d)
| summarize SignIns = count(), Apps = make_set(AppDisplayName, 10)
    by UserPrincipalName, ClientAppUsed
| top 50 by SignIns

• Decide the target scope as a group, not "All users". A group lets you stage rings and pull people out in one change.
• Write down the rollback: which policy ID, what state to set, who can do it (Conditional Access Administrator via PIM) and how they get in if their own sign-in is affected.

#2. Create the policy in report-only

• Create or modify the policy with state set to enabledForReportingButNotEnforced. The three valid states are enabled, disabled and enabledForReportingButNotEnforced.

az rest --method patch \
  --url "https://graph.microsoft.com/v1.0/identity/conditionalAccess/policies/$POLICY_ID" \
  --body '{"state":"enabledForReportingButNotEnforced"}'

• Remember report-only does not evaluate everything the same way. Policies with a block or require compliant device control in report-only can prompt macOS, iOS and Android users for a device certificate during evaluation; check the current Microsoft guidance for your platforms before targeting them.
• Use What If in the Entra admin centre (Conditional Access > Policies > What If) for specific edge cases: a break-glass account, a service account, a guest, a user on an unmanaged device from an unknown location. What If evaluates the policies as written; it does not tell you how many real sign-ins match.

#3. Read the report-only results

Let report-only run long enough to cover weekly patterns (batch jobs, month-end, people on leave): at least 7 days for most changes.

• Who would have been blocked or challenged?

let policyName = "CA-042 Require compliant device for Office 365";
SigninLogs
| where TimeGenerated > ago(7d)
| mv-expand cap = ConditionalAccessPolicies
| where tostring(cap.displayName) == policyName
| extend r = tostring(cap.result)
| summarize SignIns = count(), Users = dcount(UserPrincipalName) by r

Report-only results you'll see include reportOnlySuccess, reportOnlyFailure, reportOnlyNotApplied and reportOnlyInterrupted. reportOnlyFailure is what would have been blocked. reportOnlyInterrupted means the user would have been asked for something more (MFA, terms of use) that they did not complete during that sign-in.

• Break the failures down by who, which app and which client:

SigninLogs
| where TimeGenerated > ago(7d)
| mv-expand cap = ConditionalAccessPolicies
| where tostring(cap.displayName) == "CA-042 Require compliant device for Office 365"
| where tostring(cap.result) in ("reportOnlyFailure", "reportOnlyInterrupted")
| summarize count() by UserPrincipalName, AppDisplayName, ClientAppUsed,
    OS = tostring(DeviceDetail.operatingSystem),
    Compliant = tostring(DeviceDetail.isCompliant)
| order by count_ desc

• Repeat the query against AADNonInteractiveUserSignInLogs. In that table ConditionalAccessPolicies may be stored as a string; wrap it with todynamic() before mv-expand if the query returns nothing.
• If the policy targets workload identities, check AADServicePrincipalSignInLogs the same way.
• For every user or app in the failure list, decide: fix the client (enrol the device, update the app), add a scoped exclusion with an owner and expiry date, or accept the block.

Warning

An exclusion added "temporarily" during rollout tends to become permanent. Record every exclusion with a ticket, an owner and a review date, and put excluded accounts in a dedicated group so they are visible.

#4. Enforce in rings

• Ring 0: IT and the change owners. Switch the policy to enabled with the include group containing only them. Wait a day.
• Ring 1: a pilot group with a mix of platforms and locations, 5–10% of users. Wait several days and watch the service desk queue.
• Ring 2: everyone in scope.
• After each ring, run the failure query again, this time on enforced results:

SigninLogs
| where TimeGenerated > ago(1d)
| mv-expand cap = ConditionalAccessPolicies
| where tostring(cap.displayName) == "CA-042 Require compliant device for Office 365"
| where tostring(cap.result) == "failure"
| summarize Failures = count() by bin(TimeGenerated, 1h), AppDisplayName
| render timechart

• Keep the rollback command ready in the change record:

az rest --method patch \
  --url "https://graph.microsoft.com/v1.0/identity/conditionalAccess/policies/$POLICY_ID" \
  --body '{"state":"enabledForReportingButNotEnforced"}'

Caution

Switching a block policy back to report-only is safe and fast. Deleting it is not: you lose the configuration, and if the policy was the only control on a sensitive app, you open it up. Roll back by state, never by delete.

#5. After enforcement

• Alert on a sudden rise in Conditional Access failures (ResultType 53003 is "blocked by Conditional Access") so a bad change shows up in minutes, not via the service desk.

SigninLogs
| where TimeGenerated > ago(15m)
| where ResultType == 53003
| summarize Blocked = count() by AppDisplayName
| where Blocked > 50

• Commit the final policy JSON next to the snapshot and diff it.
• Review the exclusion group on its review date.

#Common surprises

#Prevent it next time

• Treat Conditional Access as code: JSON in a repo, pull request review, and Graph-based deployment from a pipeline.
• Keep report-only as a mandatory stage with a minimum duration in your change template.
• Name policies with an ID and intent so KQL filters and audits stay readable.
• Test break-glass sign-in quarterly and after every major Conditional Access change.