DEV Community: Alex Mitu

The error said "Action failed." That was all it said.

Alex Mitu — Wed, 22 Jul 2026 12:57:30 +0000

A nightly sync job processes a few thousand records against a third party API. It runs fine for weeks. Then one night, one record throws Operation failed and the job stops there. No stack trace, no field name, no hint about what actually went wrong. Every other record in the batch went through without a problem.

If you've worked with any API that talks to a system you don't control, you've hit this exact wall. The error is real, the failure is real, and the message tells you almost nothing. This is not really about that one error string. It's about what to do when a vendor decides vague is safer than specific.

Vague errors are usually on purpose

It's tempting to assume the API author was just being lazy. Sometimes that's true. But a lot of the time, generic errors are a deliberate choice: security teams don't want to leak internal state through error messages, and support teams don't want users acting on assumptions that turn out to be wrong. Action failed protects the vendor from a thousand support tickets built on a misread error code.

That reasoning makes sense from where they're sitting. It doesn't make the debugging session any shorter from where you're sitting.

Step one: stop trusting the summary

Most tools that call an API don't show you the raw response. They catch it, wrap it, and print something like Job failed for item 4821. That wrapper is doing you a disservice the moment something unusual happens, because it's discarding the one thing you actually need: the body of the response the server sent back.

Before doing anything else, find where the raw HTTP response is logged, or reproduce the call yourself directly against the API:

curl -i -X POST "https://api.example.com/v1/items/4821/sync" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d @item-4821-payload.json

Nine times out of ten the raw response has more in it than the tool ever showed you. A status code, a nested error.details object, sometimes even the exact field that failed validation. The wrapper just never surfaced it.

Step two: isolate down to one unit

If the failure is buried inside a batch job, don't try to debug the batch. Pull out the single item that's failing and run it on its own. This sounds obvious written down, but under time pressure it's the step people skip, because rerunning the whole job "just to see if it happens again" feels faster than isolating.

It isn't faster. A batch run gives you noise: successful items, retries, unrelated warnings, timestamps that don't line up. A single isolated call gives you a clean before and after. Cut the batch out of the picture entirely and you can iterate in seconds instead of minutes.

If you're working from logs instead of live access, the same principle applies. Grep for the specific identifier, not the general error string. Searching for Operation failed gets you a wall of matches. Searching for the one ID that's actually broken gets you a narrow, readable slice.

grep -n "4821" service.log | grep -v "heartbeat"

Step three: resolve the opaque ID before anything else

A lot of "generic error, specific cause" situations turn out to be an identity problem in disguise. The ID in your log line is often internal, an integer, a GUID, a hash, something that means nothing to a human and nothing to the API's own documentation search. Before you theorize about causes, translate that ID into something you can actually reason about: a name, a path, a resource type.

Most APIs that use opaque IDs also expose a lookup endpoint for exactly this purpose. It's easy to skip this step because it feels like a detour from the "real" debugging. It isn't. Half the time, once you know that ID 4821 is actually "a shared mailbox with no owner" or "a file with a name 400 characters long," the cause explains itself.

Step four: write the raw response down, not your interpretation of it

Once you get the real response back, resist the urge to summarize it in your own words right away. Copy the actual payload into your notes first. Interpretations drift. Two days later you'll remember "something about permissions" when the actual field said insufficient_scope: files.write, and those are not the same debugging path.

This matters even more if you're about to hand the case to someone else, or escalate it. "It's probably a permissions issue" gets deprioritized. A pasted error body with the exact scope name attached gets picked up fast, because the next person doesn't have to reproduce your work to trust it.

What this actually is

None of this is specific to any one API or platform. It's the same four moves every time: stop trusting the wrapper, isolate to one unit, resolve the opaque identifier, and preserve the raw evidence before you translate it into a theory. The tools change. The order doesn't.

The annoying part is that this process feels slow the first time you do it on a new system, and fast every time after. Once you know where a particular API hides its real error body, or which lookup endpoint turns an ID into a name, the next generic failure takes you five minutes instead of an afternoon. That's really the whole payoff: you're not getting better at guessing, you're building a small map of exactly where each system likes to hide the truth.

So next time something throws you a wall of nothing, resist the urge to just retry it and hope. The information is almost always there. It's just sitting one layer below where the error message stopped looking.

Why Your EWS Impersonation Suddenly Stopped Working (And It's Probably Not Throttling)

Alex Mitu — Fri, 10 Jul 2026 18:17:12 +0000

Two months ago I picked up a ticket that looked routine: a job that reads mailbox data from Microsoft 365 through EWS, running fine for over a year, started failing on a subset of mailboxes in one tenant. Same app registration, same code path, same service account. The error in the logs pointed at throttling, so that's where the admin had already spent three days looking. Wrong direction. The actual cause had nothing to do with throttling budgets.

This mix-up happens constantly right now, and it's worth understanding why, because the fix for one problem does nothing for the other, and chasing the wrong one wastes days.

TL;DR: if your EWS failures don't scale with request volume, stop tuning throttling and go check your Application Access Policy scope groups instead.

What EWS throttling actually looks like

Exchange Online throttles EWS the same way it always has: budget-based. Every account gets a policy (the default is EwsDefaultThrottlingPolicy, but plenty of tenants layer custom ones on top) that tracks a slowly-refilling budget rather than a simple call count. When you overspend it, you get back a 503 or 429 with an X-MS-Diagnostics header telling you which budget got exhausted, usually the connection count or the concurrent-request limit.

The tell for real throttling is consistency. It scales with load, correlates with concurrency and batch size, and clears up within minutes once you back off. If you graph failure rate against request volume, you'll see a clean relationship. If you double your batch size, failures increase. If you throttle yourself proactively (respecting Retry-After, staying under EWSFindCountLimit for FindItem calls), it mostly goes away.

That correlation is the whole diagnostic test. If your failures don't scale with volume, you're not looking at a throttling problem, no matter what the error message on the surface says.

What actually changed

Over the past year or so, Microsoft tightened enforcement in two places that both produce errors easy to mistake for throttling.

First, Application Access Policies. New-ApplicationAccessPolicy has existed for years, letting admins scope which mailboxes an app registration can touch instead of granting it impersonation rights over the entire tenant. It used to be optional, something only security-conscious tenants bothered configuring. That's changed. More tenants are applying scoped access policies as a default hardening step, often as part of a broader Conditional Access or Secure Score push, and they're doing it without touching the app's actual permissions or the service account's RBAC role. The app still has ApplicationImpersonation. It just can't reach mailboxes outside its assigned scope group anymore, and mailbox group membership drifts constantly as people get added, removed, or reassigned.

Second, the long tail of Basic Auth deprecation. Most environments moved off Basic Auth for EWS years ago, but I still see tenants with a leftover service principal or a scheduled task somewhere using a cached token flow that quietly stopped working when Microsoft closed another enforcement gap. It's rarely the primary auth path failing outright. It's usually one forgotten integration that everyone assumed was already migrated.

Both of these produce intermittent, mailbox-specific failures. Neither has anything to do with request volume.

Telling them apart

Here's the sequence I actually use now before touching any throttling knobs.

1. Check whether failures correlate with specific mailboxes, not specific times. Pull the failing mailbox list and check it against the policy's scope group:

Get-ApplicationAccessPolicy | Format-List AppId, PolicyScopeGroupId, AccessRight

Get-DistributionGroupMember -Identity "EWS-Scoped-Mailboxes" |
    Select-Object PrimarySmtpAddress

Compare the mailbox list this returns against the mailboxes that are actually failing. Anything missing from the group is out of scope, and that's your answer.

2. Read the actual error code, not just the HTTP status. These two failures look almost identical if your logging only captures the status code:

# Real throttling
HTTP/1.1 503 Service Unavailable
X-MS-Diagnostics: 3006;reason="EWSMaxConcurrency budget exhausted";error_category="transient"

# Access policy scoping
HTTP/1.1 403 Forbidden
<S:Fault>
  <S:Code><S:Value>Sender</S:Value></S:Code>
  <S:Reason><S:Text>ErrorAccessDenied</S:Text></S:Reason>
</S:Fault>

If your logging layer only captures the HTTP status and not the response body, fix that first. You're throwing away the one piece of information that tells these two apart.

3. Check the Microsoft 365 audit log around the date the failures started, not the date they were noticed:

Search-UnifiedAuditLog -StartDate (Get-Date).AddDays(-30) -EndDate (Get-Date) `
    -Operations "New-ApplicationAccessPolicy","Set-ApplicationAccessPolicy","Remove-ApplicationAccessPolicy" |
    Select-Object CreationDate, UserIds, Operations

I've found the actual change was made three weeks before anyone noticed, during a routine security review nobody connected to this app.

4. Confirm the app's permission model.

Get-ManagementRoleAssignment -RoleAssignee "svc-backup-app" |
    Where-Object { $_.Role -eq "ApplicationImpersonation" }

If this returns a result, the app can impersonate any mailbox in the org by design, unless an access policy is actively narrowing that down. Worth knowing before you assume the access policy is even the right suspect.

If none of that turns anything up and failures genuinely track with request volume and clear up after backing off, then fine, it's real throttling, go tune your batching.

Fixing the actual problem

If it's an access policy issue, the fix is usually mechanical but tedious: audit the scope group, figure out why the mailbox fell out of it, and set up something that doesn't rely on someone remembering to update a static group every time a mailbox changes hands. A dynamic distribution group tied to whatever attribute actually determines "should this app reach this mailbox" beats a manually maintained list every time:

New-DynamicDistributionGroup -Name "EWS-Scoped-Mailboxes" `
    -RecipientFilter {RecipientTypeDetails -eq "UserMailbox" -and CustomAttribute1 -eq "backup-enabled"}

Point the access policy at this group instead of a static one. Membership now updates itself whenever CustomAttribute1 changes, instead of depending on someone remembering to edit a list.

The bigger fix, though, is getting off tenant-wide impersonation entirely. Microsoft has been pushing RBAC for Applications as the replacement, and it's worth the migration even outside of this specific problem. Instead of one service account impersonating anyone, you assign granular, resource-scoped permissions directly to the app registration:

New-ServicePrincipal -AppId <your-app-id> -ObjectId <object-id> -DisplayName "Backup-App"

New-ManagementScope -Name "Backup-App-Scope" `
    -RecipientRestrictionFilter {CustomAttribute1 -eq "backup-enabled"}

New-ManagementRoleAssignment -Role "Application Mail.Read" -App <your-app-id> `
    -CustomRecipientWriteScope "Backup-App-Scope"

This replaces a single tenant-wide ApplicationImpersonation grant with a role scoped to exactly the mailboxes the management scope defines. Exact cmdlet parameters shift a bit between Exchange Online PowerShell module versions, so treat this as the shape of the migration and confirm current syntax against Microsoft's docs before running it against production. It's more setup work up front, but it means you stop being exposed to "someone tightened a policy somewhere and now half your mailboxes silently fail" as a category of incident. Given that EWS itself is on Microsoft's retirement roadmap in favor of Graph API, this is also the direction you'd be migrating toward anyway. Doing the permission model migration first, before the API migration, splits the work into two smaller, less risky changes instead of one large one.

One more thing worth doing regardless: separate your retry logic by error type. A lot of applications treat every failed EWS call the same way and retry with exponential backoff, which is correct for a 503 throttling response and actively harmful for a 403 access-denied response:

def should_retry(status_code, error_code):
    if status_code == 503 or error_code == "ErrorServerBusy":
        return True   # transient, back off and retry
    if status_code in (401, 403) or error_code == "ErrorAccessDenied":
        return False  # a permission problem, retrying won't fix it
    return False

Retrying an access-denied error just burns your throttling budget on calls that were never going to succeed, and can end up creating the throttling problem you originally thought you had.

The pattern underneath this

None of this is exotic. It's the same lesson as always: a generic error handler will happily lump together two unrelated failure modes under one log line, and the fix for each one does nothing for the other. The throttling ticket that isn't actually about throttling has become common enough in the last year that I've started assuming access policy scope first and traffic volume second, and I've been right more often than not.