DEV Community

yutianle
yutianle

Posted on

When the Vendor Is Down and Your Tenant Is Still Exposed: Ordering Remediation Work

#remediation #prioritisation #microsoft365 #msp

When the Vendor Is Down and Your Tenant Is Still Exposed: Ordering Remediation Work

An incident with a third-party SaaS vendor produces two problems at once, and they compete for the same people. The service you depend on is suspended, so operations need attention now. The credential that opened your tenant may still be usable elsewhere, so security work is urgent too.

In the ConnectWise Cloud Backup case both conditions held. Cloud Backup, SaaS Security and Cloud Management were shut down on 17 and 18 October 2026, affecting scheduled backups, archive access, restores, monitoring and licence management. Simultaneously, partners were told to verify that the actor had not changed configuration or established persistence in their Microsoft 365 tenants.

This article orders that work. The ordering follows from what the public evidence actually showed rather than from a generic checklist.

Priority one: credentials the actor created

Huntress reported that in one tenant the Cloud Management application for Microsoft Teams added itself to Global Administrator, then ran a Python script creating new client secrets on an MSP management application and on a security vendor's application.

Newly created secrets outlive the block on the original credential. Remediation therefore starts by enumerating every client secret and certificate on enterprise applications and identifying material created during the intrusion window. Anything unexplained is revoked. This is the only step in the list where delay has an obvious mechanical consequence: a live secret keeps working.

Priority two: role assignments held by applications

A service principal in Global Administrator is a standing privilege escalation waiting for a credential. Review assignments to Global Administrator and Privileged Role Administrator, and confirm whether each assignee is a person or an application. Where an application holds a role it does not need, remove it before investigating anything else about that application.

This step also addresses the possibility that role changes were made in tenants where the log review has not yet reached. The current state is checkable in minutes; the history takes longer.

Priority three: consent grants, with an owner for each

Every vendor application with tenant-wide consent is a durable grant that survives password resets. Build the list, attach a business owner, and record the date consent was given. Grants nobody can explain are the ones to revoke first.

In practice this produces the first honest map of dependency the organisation has had. It also answers the operational question raised by the outage: which functions stop if this vendor is unavailable.

Priority four: log review against two filters

The timeline in the public record gives the filters. Application sign-ins from 144.172.64[.]128 on 15 October between 06:13 and 08:34 UTC cover the reported Graph token acquisition. A second filter on service principal sign-ins regardless of source address catches the role change, which did not originate from that address.

Running only the IP filter is the common mistake, because the IP is the memorable artefact and it is also the incomplete one.

Priority five: continuity, not detection

Restores and archive access were suspended with no published restoration timeline. Someone needs to confirm that the organisation can survive that gap: whether an alternate backup exists, whether the retention period on it is adequate, and whether the restore procedure has been tested.

This is unglamorous work and it is what the outage actually imposed. ConnectWise stated that customer backups remained encrypted and protected, and that it found no evidence of unauthorised access to them, which supports treating this item as availability rather than confidentiality.

What outside measurement contributes

Very little to the remediation sequence, and it is worth being explicit about why. Queries run on 28 October 2026 returned 170 records for domain="connectwise.com", 76 for the same domain restricted to port 443, and 0 for ip="144.172.64.128". Those figures describe the vendor's published estate and the absence of an indexed host at the attacker address. None of them tells an administrator which of their applications holds a dangerous grant.

Where a count does help is in the dependency conversation. A dated baseline of the vendor's observable footprint is a small piece of evidence in the argument that a single external party carries a large share of operational continuity.

Limits

The priorities above derive from one investigator's published findings and the vendor's own guidance. Tenants that never held the affected service principal had a much smaller problem, and their work starts lower down the list. No public source establishes that any customer's backup data was accessed, so remediation here is about closing paths and confirming absence of change rather than about confirmed loss.

References

Top comments (0)