Mike Anderson

Posted on Jul 14

ICS Cybersecurity A–Z (Part 2): Operations, Incident Response, and Threat Modeling

#cybersecurity #security #architecture #iot

Part 1 covered the foundation: segmentation, hardening, monitoring, and safe assessment.

That is where many ICS security programs stop.

They build a good diagram, deploy a monitoring tool, fix some obvious gaps, and then assume the environment is secure.

It is not.

An ICS security program only becomes real when the organization can operate it every day, detect abnormal behavior, respond without harming the process, recover from compromise, and explain risk clearly to leadership.

Part 2 is about that operating layer.

We will cover:

Steady-state OT cyber operations
Patch, change, access, and backup discipline
Incident response for control environments
Large-scale or nation-state campaign preparation
Legal and operational boundaries for active defense
Threat modeling with MITRE ATT&CK for ICS
Metrics that matter to leadership
A practical 30/60/90-day improvement plan

The goal is simple:

Keep essential services running while reducing the chance that a cyber event becomes a safety, reliability, or public-impact event.

1. Steady-state OT cyber operations

Security operations in ICS must be predictable.

The worst OT security programs are reactive. They patch after a breach, block traffic during an outage, or discover vendor accounts only after suspicious access appears.

A mature program has an operating rhythm.

Operating rhythm

Daily

Activity: review high-severity OT alerts and remote access activity.
Owner: SOC / OT security.
Evidence: alert notes and triage decisions.

Weekly

Activity: review new assets, new network flows, failed logins, and monitoring gaps.
Owner: OT security / network team.
Evidence: asset deltas and flow review.

Monthly

Activity: review privileged and vendor accounts.
Owner: IAM / OT owner.
Evidence: access review sign-off.

Monthly

Activity: validate backup job status and offline backup inventory.
Owner: OT engineering.
Evidence: backup report and sample restore evidence.

Quarterly

Activity: review firewall rules and segmentation exceptions.
Owner: network / OT security.
Evidence: rule review export and exception decisions.

Quarterly

Activity: patch planning and vendor advisory review.
Owner: OT engineering / vendor manager.
Evidence: patch plan and risk acceptance.

Semiannual

Activity: restore test for critical HMI, historian, and PLC logic backups.
Owner: OT engineering.
Evidence: restore test results.

Annual

Activity: OT tabletop exercise and incident response test.
Owner: CISO / plant leadership.
Evidence: exercise report and improvement plan.

This rhythm matters because ICS risk accumulates quietly.

A temporary vendor account becomes permanent. A firewall exception created during commissioning is never removed. A spare HMI misses patches for two years. A controller is replaced but never added to inventory.

The operating model catches these before they become attack paths.

2. Patch and change management

Patching in OT is not the same as patching laptops.

You still need vulnerability management, but the process must account for vendor support, process uptime, safety impact, and rollback.

A practical OT patch process:

Track vendor advisories and CISA ICS advisories for products in your environment.
Identify affected assets from the OT inventory.
Classify risk by exploitability, exposure, process criticality, and compensating controls.
Test patches on a spare, lab system, or non-critical asset first.
Confirm vendor support for the patch level.
Schedule the change with operations.
Confirm backups and rollback.
Apply during an approved window.
Monitor the asset and process after the change.
Retain evidence.

Patch priority should not be based only on CVSS.

For ICS, prioritize:

Internet-exposed OT services.
Remote access infrastructure.
Engineering workstations.
HMIs with broad process visibility.
Historians bridging IT and OT.
Vulnerabilities with known exploitation.
Vulnerabilities that allow authentication bypass, remote code execution, or unauthorized control modification.
Assets with weak segmentation or no compensating controls.

A medium CVSS issue on a directly reachable HMI may matter more than a critical issue on an isolated spare asset.

Change control is a security control

Every OT change should answer:

What is changing?
Which process could be affected?
Who approved it?
What is the rollback plan?
What evidence will prove the change was successful?
What monitoring is required after the change?
What is the stop condition?

This applies to firewall rules, controller logic, HMI software, remote access, patches, sensor changes, and vendor maintenance.

Poor change control is one of the most common causes of self-inflicted OT incidents.

3. Account and access hygiene

Identity is often the weak point in OT.

Many environments still have shared operator accounts, old vendor accounts, unmanaged local administrators, and remote access paths that were created for commissioning and never removed.

Minimum controls:

MFA for remote access and privileged access where supported.
Unique named accounts for engineers and vendors.
No standing vendor access unless formally justified.
Monthly vendor account review.
Quarterly privileged access review.
Break-glass accounts protected, monitored, and tested.
Password vaulting for shared emergency credentials.
Session recording for privileged remote access.
Disable accounts immediately when engineers, contractors, or vendors leave.
Restrict engineering tool access to approved workstations.

The real risk is not only credential theft. It is loss of accountability.

If five people use the same engineering account, you cannot reliably determine who changed a PLC program, acknowledged an alarm, or exported a project file.

4. Backup and recovery testing

Backups are not a compliance artifact in ICS. They are a control for public safety and service continuity.

You need backups for:

HMI images
Engineering workstation images
SCADA server configuration
Historian configuration
PLC and RTU logic
Network device configuration
Firewall rules
Switch configuration
Remote access gateway configuration
License keys and vendor installation media
Offline documentation needed for recovery

The minimum standard is:

Keep offline or immutable copies.
Keep at least one copy physically or logically disconnected.
Test restores, not only backup completion.
Validate PLC logic backups on spare hardware where possible.
Record firmware and software version dependencies.
Store recovery procedures where they are accessible during an IT outage.
Include operations in recovery drills.

The question leadership should ask is not:

"Do we have backups?"

The better question is:

"Can we restore the most critical process-control function from known-good backups under incident conditions?"

5. OT incident response: containment without causing harm

Traditional incident response often says isolate the host quickly.

That may be right for an office laptop.

It may be dangerous for a controller, HMI, historian, safety system, or engineering workstation involved in a live process.

OT incident response must be safety-led.

Preparation

Before an incident, create:

OT incident response plan.
Plant contact roster.
OT asset inventory with process criticality.
Network diagrams and trust boundaries.
Communication plan if email and VoIP are down.
Forensic collection procedure.
Escalation path to safety, legal, privacy, executives, and sector authorities.
Pre-approved containment options.
Emergency change process.
Out-of-band communication method.
Tabletop exercise schedule.

Triage questions

When an alert fires, ask:

Which asset is affected?
What physical process does it support?
Is the process stable?
Is this read traffic, write traffic, authentication activity, or logic change?
Is the activity inside an approved change window?
Is a vendor currently authorized to connect?
Can containment disrupt safety or availability?
What evidence must be preserved before action?
Who has authority to approve containment?
What rollback is available?

These questions prevent the SOC from making a technically correct but operationally unsafe decision.

Severity model

SEV-1 Critical

OT example: active unauthorized control action, ransomware affecting operations, confirmed manipulation of controller logic, or safety impact.
Response expectation: activate crisis management, OT IR, legal, executive leadership, and the safety owner.

SEV-2 High

OT example: compromised engineering workstation, unauthorized PLC write attempt, vendor account compromise, or malware on HMI.
Response expectation: immediate OT/security bridge and containment plan approved by the OT owner.

SEV-3 Medium

OT example: suspicious scan, failed logins, unauthorized device detected, or policy violation.
Response expectation: same-day investigation and corrective action.

SEV-4 Low

OT example: false positive, benign misconfiguration, or informational monitoring gap.
Response expectation: track and tune through the normal backlog.

Containment options

Containment should be targeted.

Containment decision guide

Suspicious laptop in OT VLAN

Safer option: disable or quarantine the switch port for that endpoint.
Avoid unless approved: shutting down the entire switch.

Enterprise-to-OT attack path

Safer option: block the enterprise-side route or disable the jump path.
Avoid unless approved: disrupting Level 2 or Level 1 communications.

Compromised vendor account

Safer option: disable the account and terminate the active session.
Avoid unless approved: broad password resets during live operations without a plan.

Unauthorized PLC write source

Safer option: block the source IP at the cell firewall or switch ACL.
Avoid unless approved: remote stop command to PLC.

Malware on HMI

Safer option: move operator function to standby HMI, then isolate the affected host.
Avoid unless approved: killing the HMI process during active operation.

Suspected logic change

Safer option: compare logic to known-good backup and involve a controls engineer.
Avoid unless approved: reloading logic without process validation.

The principle is:

Isolate the attacker, not the process.

Eradication and recovery

Recovery actions may include:

Reimage HMIs or engineering workstations from golden images.
Reload controller logic from known-good backups.
Validate checksums or vendor integrity indicators.
Reset credentials in the affected zone.
Patch the entry point.
Review firewall and remote access logs.
Monitor for recurrence.
Reconnect enterprise paths only after the route of compromise is understood and controlled.

Do not reconnect because "production needs it" without understanding the attack path. That is how reinfection happens.

Post-incident review

Within 72 hours for major incidents, document:

Timeline.
Initial access vector.
Affected assets.
Physical process impact.
Containment decisions and approvals.
Evidence collected.
Root cause.
Controls that worked.
Controls that failed.
Detection gaps.
Remediation owners and due dates.
Leadership summary.

The best incident reviews are not blame exercises. They are control improvement sessions.

6. Preparing for large-scale or nation-state campaigns

Some OT incidents are not isolated.

Utilities, transportation, energy, manufacturing, and public-sector operators may face coordinated campaigns involving destructive malware, credential attacks, vendor compromise, DDoS, influence operations, and physical security pressure.

The objective during a large-scale campaign is not perfect investigation.

The objective is continuity of essential service, safe operation, evidence preservation, and coordinated defense.

Before the campaign

Prepare:

Sector threat intelligence relationships.
Membership or contact path with relevant ISAC/ISAO.
National CERT reporting path.
Pre-approved emergency firewall rules.
Pre-approved remote access shutdown procedure.
Emergency vendor contact list.
"Island mode" or isolation plan where operationally feasible.
Manual operation procedures.
Spare hardware and offline media.
Out-of-band communications.
Executive decision matrix.

Emergency isolation must be practiced. A plan that exists only in a PDF is not a plan.

During the campaign

Actions may include:

Activate the OT incident response bridge.
Confirm process stability with operations.
Increase monitoring on remote access, firewalls, DNS, identity, and OT sensors.
Disable non-essential vendor access.
Restrict internet-facing OT DMZ services.
Apply pre-approved blocks for confirmed malicious infrastructure.
Validate backups and standby systems.
Monitor for unauthorized engineering protocol use.
Share indicators with your ISAC or national CERT.
Preserve evidence for law enforcement and sector response.

Be careful with blanket actions.

"Force every privileged password change immediately" may be appropriate in some cases, but in OT it must be planned so you do not lock out operators, break services, or lose access to legacy systems during a crisis.

Defensive countermeasures

Private organizations can and should defend their own environment.

Appropriate defensive actions include:

Blocking known malicious IPs and domains at your perimeter.
Sinkholing malicious domains inside your own DNS environment when properly authorized.
Null-routing attacker infrastructure at your boundary.
Disabling compromised accounts.
Quarantining affected endpoints.
Deploying honeytokens or decoy shares.
Increasing detection sensitivity for known TTPs.
Sharing indicators with trusted sector partners.
Supporting takedown efforts through proper legal and provider channels.

No hack back

Private-sector teams should not conduct offensive retaliation.

Do not access attacker infrastructure, modify external systems, steal data back, or launch counterattacks.

The right role is:

Stop the attack inside your boundary.
Preserve forensic evidence.
Share indicators quickly.
Support law enforcement, CERT, regulators, and sector response bodies.
Maintain safe operations.

That is how private organizations contribute to national defense without creating legal, diplomatic, or operational risk.

7. Threat modeling: prioritize what actually reduces risk

Threat modeling prevents random security spending.

For ICS, use the process and the attack path together.

A practical model includes:

Critical process.
Assets supporting that process.
Trust boundaries.
Possible attacker entry points.
Attack path from IT or remote access to control impact.
Existing controls.
Detection coverage.
Response options.
Control gaps.
Remediation owner.

MITRE ATT&CK for ICS is useful because it maps adversary behavior to ICS-specific tactics and techniques.

Example attack path:

Example attack path

Initial access

Example behavior: vendor VPN account compromised.
Expected telemetry: VPN login from unusual location and MFA anomaly.
Required control: MFA, conditional access, and vendor access approval.
Owner: IAM / OT security.

Discovery

Example behavior: attacker scans the OT subnet.
Expected telemetry: new network flows, protocol discovery, and sensor alert.
Required control: segmentation, passive monitoring, and blocked routes.
Owner: network / SOC.

Lateral movement

Example behavior: RDP to engineering workstation.
Expected telemetry: jump host logs and Windows logon events.
Required control: jump host only, no direct RDP, and named accounts.
Owner: infrastructure.

Collection

Example behavior: project files copied.
Expected telemetry: file access logs and unusual archive creation.
Required control: least privilege, monitoring, and data access review.
Owner: OT engineering.

Control manipulation

Example behavior: unauthorized PLC write or logic change.
Expected telemetry: engineering protocol write, controller event, and logic checksum change.
Required control: PLC ACL, change workflow, and alerting.
Owner: OT engineering.

Inhibit response

Example behavior: alarms suppressed or HMI altered.
Expected telemetry: HMI configuration change and alarm state changes.
Required control: HMI change control, backups, and monitoring.
Owner: operations.

This model tells you where to spend money.

If the highest-risk path is vendor VPN to engineering workstation to PLC logic change, then buying another generic IT scanner is not the first priority.

Better priorities may be:

Tighten vendor access.
Remove direct RDP.
Add engineering workstation allowlisting.
Monitor engineering protocol writes.
Implement PLC logic backup and checksum validation.
Exercise the containment process.

8. Metrics leadership should care about

Leadership does not need every alert.

They need a clear view of operational cyber risk.

Good OT security metrics:

Percentage of critical OT assets inventoried: shows whether the team can defend what matters.
Number of direct IT-to-OT flows: measures segmentation risk.
Vendor accounts active outside approved windows: measures third-party access risk.
Percentage of critical HMIs and engineering workstations with tested backups: measures recovery confidence.
Number of unauthorized or unexplained OT protocol write attempts: measures control-layer threat activity.
Critical OT vulnerabilities past SLA with no compensating control: measures unresolved exposure.
Percentage of firewall rules reviewed in the last quarter: measures control hygiene.
Mean time to triage high-severity OT alerts: measures SOC readiness.
Number of successful restore tests: measures resilience.
Open exceptions by age and criticality: measures risk debt.

The best board-level statement is not "we deployed an OT monitoring tool."

A better statement is:

"We have identified 96% of critical OT assets, removed all direct enterprise-to-controller access, reviewed 87% of OT firewall rules this quarter, and validated restore procedures for the three most critical process-control functions. Remaining risk is concentrated in two legacy PLC families and one vendor access path, with remediation owners assigned."

That is operationally meaningful.

9. 30/60/90-day improvement plan

If you are starting from a messy environment, do not try to fix everything at once.

First 30 days: establish visibility and stop obvious risk

Build critical OT asset inventory.
Identify direct IT-to-OT paths.
Review remote vendor access.
Confirm backups exist for critical HMI, SCADA, and PLC assets.
Collect firewall rules and network diagrams.
Start passive monitoring in one critical segment.
Identify default credentials and shared accounts.
Define incident contact roster.

Days 31–60: enforce control points

Remove direct enterprise access to controllers.
Move vendor access behind MFA and jump host.
Review and reduce broad firewall rules.
Implement logging for remote access and jump hosts.
Validate backups through sample restore.
Create initial OT detection use cases.
Establish change approval for PLC logic and firewall changes.
Create exception register.

Days 61–90: operationalize

Run an OT tabletop exercise.
Test incident containment decision flow.
Review privileged access.
Tune monitoring detections.
Build threat model for the most critical process.
Assign remediation backlog by risk.
Report metrics to leadership.
Schedule quarterly control reviews.

This sequence creates measurable progress without destabilizing operations.

Final takeaway

ICS cybersecurity is not about adding IT controls blindly to plant environments.

It is about protecting the physical process by controlling access, reducing unsafe paths, monitoring meaningful behavior, and responding with operational discipline.

The strongest OT programs share a few traits:

They know their assets.
They respect safety and availability.
They segment based on process risk.
They control vendor access.
They monitor controller-relevant behavior.
They test backups.
They practice incident response.
They use threat modeling to prioritize.
They can explain residual risk to leadership.

Start there.

Keep the water flowing, the lights on, and the process under control.

DEV Community