Self-Healing Networks: EEM Applets That Fix Problems Before You Wake Up

#cisco #networking #automation #eem

Self-Healing Networks: EEM Applets That Fix Problems Before You Wake Up

This is Part 2 of a 5-part series on Cisco EEM. In Part 1 we covered the fundamentals — event plus action — and built three applets across three trigger types. Now we put that model to work on the thing everyone actually wants: a network that fixes itself.

There's a particular kind of 3 AM page that stings the most: the one for a problem that would have cleared on its own if something had just nudged it. A port that err-disabled and only needed a clear. An adjacency that dropped and would have come right back if the interface had been bounced. A process chewing CPU that a restart would have settled. You wake up, VPN in, run one command, and go back to bed — having added nothing a well-written applet couldn't have done at 3:00:04.

Self-healing is EEM's most satisfying use case, and its most dangerous. Done well, it turns a category of pages into log entries you read over coffee. Done carelessly, it builds a machine that hides failures, masks root causes, and occasionally makes outages worse by thrashing. This part is about doing it well — the applets, and just as importantly, the discipline that keeps self-healing from turning into self-harm.

The mindset: heal the symptom, surface the cause

Before any config, internalize the one principle that separates good self-healing from bad: an applet that remediates should never remediate silently. Every automated fix must leave a loud, structured trail — because the moment you automate away a symptom, you also automate away the signal that something is wrong. If a port err-disables once a month and your applet quietly recovers it, you'll never know you have a failing cable until it fails completely.

So the pattern for every applet in this article is the same three-beat rhythm:

Detect the condition.
Remediate it — but with limits.
Announce loudly what happened, so a human can see the pattern even though they didn't have to act on the instance.

Self-healing isn't "make the problem invisible." It's "handle the instance automatically, but make the pattern impossible to miss." Hold onto that as we build.

Applet 1: Err-disable recovery that's actually safe

The err-disable recovery applet is the "hello world" of self-healing, and it's usually done badly — blindly clearing every err-disabled port regardless of why it went down. Some err-disable causes (a bad transceiver, a loop) should recover automatically. Others (a security violation, BPDU guard tripping) are telling you something you need to know, and auto-clearing them is exactly wrong.

The native IOS feature already lets you scope auto-recovery per cause, and for the causes you trust, it's the cleaner tool:

errdisable recovery cause link-flap
errdisable recovery cause bpduguard
errdisable recovery interval 300

But native recovery is silent — it just brings the port back with no fanfare. That's the gap EEM fills: wrap the recovery in detection and announcement so you get the pattern. Here's a version that recovers, but logs the cause and the port loudly so a recurring problem surfaces:

event manager applet ERRDISABLE-HEAL
 event syslog pattern "%PM-4-ERR_DISABLE" ratelimit 30
 action 1.0 regexp "([Gg]i[a-zA-Z0-9/]+|[Ee]t[a-zA-Z0-9/]+)" "$_syslog_msg" match iface
 action 2.0 syslog priority warnings msg "EEM SELF-HEAL: err-disable on $iface - recovering, INVESTIGATE if recurring"
 action 3.0 cli command "enable"
 action 4.0 cli command "show interfaces status err-disabled"
 action 5.0 cli command "clear errdisable interface"
 action 6.0 syslog priority notifications msg "EEM SELF-HEAL: err-disable recovery completed for $iface"

What makes this safe rather than reckless:

It captures which port (action 1.0 pulls the interface name out of the syslog message with a regex), so the announcement names the actual device that's misbehaving.
The warning-level message calls for investigation — it doesn't pretend nothing happened. If $iface shows up in that log line every night, you have a hardware problem to chase, and now you'll see it.
The ratelimit 30 stops a genuinely broken port from being cleared in a tight loop, which would generate log noise and mask the severity.

When NOT to use it: if the err-disable cause is a security violation (port-security, DHCP snooping, BPDU guard in a place it matters), you may want the port to stay down. Consider scoping the applet — or the native recovery causes — to exclude security events, so self-healing never overrides a security control.

Applet 2: Reacting to runaway CPU — capture, then calm

High CPU is the classic transient. A process spikes, the box gets sluggish, and by the time you're paged and connected, the CPU is back to normal and you have nothing to look at. Self-healing here has two jobs, and the order matters: capture the evidence first, then attempt to calm the condition. If you remediate before capturing, you've destroyed the very data you need to find root cause.

EEM can trigger on CPU via SNMP threshold. This applet fires when CPU crosses a high-water mark, snapshots the process table to a timestamped file, and then logs for escalation:

event manager applet CPU-SPIKE-CAPTURE
 event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.6.1 get-type exact entry-op ge entry-val 85 poll-interval 10
 action 1.0 syslog priority warnings msg "EEM SELF-HEAL: CPU >85% detected - capturing diagnostics"
 action 2.0 cli command "enable"
 action 3.0 cli command "show processes cpu sorted | append flash:cpu-spike.txt"
 action 4.0 cli command "show processes cpu history | append flash:cpu-spike.txt"
 action 5.0 cli command "show logging | append flash:cpu-spike.txt"
 action 6.0 syslog priority critical msg "EEM SELF-HEAL: CPU spike captured to flash:cpu-spike.txt - review required"

Notes on the mechanics:

The OID 1.3.6.1.4.1.9.9.109.1.1.1.1.6.1 is the 5-second CPU utilization from CISCO-PROCESS-MIB — adjust the index for your platform, or use the 1-minute average OID if 5-second is too twitchy.
poll-interval 10 means EEM polls every 10 seconds. Tighter polling catches shorter spikes but costs a little CPU itself — don't set it absurdly low.
Notice this applet captures but doesn't kill anything. That's deliberate. Automatically restarting a process or a module to "fix" CPU is aggressive and can cause a worse outage than the spike. For most environments, capture + escalate is the right level of self-healing for CPU; leave the actual remediation to a human who now has the evidence waiting. (In Part 3 we go much deeper on this capture-in-the-moment pattern.)

If you do have a specific, well-understood runaway that a known action resolves — and only then — you can add a remediation step. But the safe default is: heal by capturing, not by swinging a hammer.

Applet 3: Reviving a dropped routing adjacency

A flapping or dropped OSPF/BGP/EIGRP adjacency is often self-clearing, but sometimes it wedges — the neighbor is reachable, yet the session sits stuck. A carefully scoped applet can nudge it while making very clear it did so. This is where self-healing needs the most restraint, because bouncing routing sessions is not a casual act.

Here's a conservative example: on repeated OSPF adjacency loss to a specific neighbor, capture state and clear the process once, rate-limited hard so it can never become a reconvergence storm:

event manager applet OSPF-ADJ-HEAL
 event syslog pattern "%OSPF-5-ADJCHG.*FULL to DOWN" ratelimit 300
 action 1.0 syslog priority warnings msg "EEM SELF-HEAL: OSPF adjacency lost - capturing then nudging"
 action 2.0 cli command "enable"
 action 3.0 cli command "show ip ospf neighbor | append flash:ospf-heal.txt"
 action 4.0 cli command "show ip route ospf | append flash:ospf-heal.txt"
 action 5.0 wait 15
 action 6.0 cli command "clear ip ospf process" pattern "\[no\]"
 action 7.0 cli command "yes"
 action 8.0 syslog priority critical msg "EEM SELF-HEAL: OSPF process cleared after adjacency loss - INVESTIGATE root cause"

Read the guardrails carefully, because this applet is the riskiest in the article:

ratelimit 300 — at most once every 5 minutes. Clearing OSPF causes a reconvergence; doing it in a loop would be catastrophic. The hard rate limit is non-negotiable here.
wait 15 before acting — gives the adjacency a chance to recover on its own first. You only want to intervene if it's genuinely stuck, not if it's mid-recovery.
clear ip ospf process is a big hammer. In many designs you'd prefer a far gentler nudge — bouncing a single interface, or clearing a single BGP neighbor with clear ip bgp <neighbor> soft instead of the whole process. Match the remediation to the blast radius you can tolerate. For BGP specifically, soft clears avoid tearing the session down entirely.
The critical-severity announcement makes it impossible to ignore that automation touched routing. This should page someone after the fact even though it self-healed, because a wedged adjacency is a real problem you need to root-cause.

Honestly, for many shops the right version of this applet doesn't clear anything — it just captures state and escalates, exactly like the CPU applet. Auto-clearing routing is a decision to make consciously, per design, not a default to reach for.

Applet 4: Interface flap containment

Sometimes the healthiest self-healing action is to stop something rather than restart it. A port flapping rapidly (bad cable, dying SFP, duplex mismatch) generates churn — MAC moves, SPT recalculations, log floods — that can ripple outward. If a port flaps more than N times in a short window, the safest move can be to administratively shut it and hold it down, converting an intermittent problem that destabilizes the network into a clean, contained, obvious failure.

This is the native err-disable-on-flap behavior, but EEM lets you add your own logic and, crucially, a loud explanation:

event manager applet FLAP-CONTAIN
 event syslog pattern "%LINK-3-UPDOWN.*Interface GigabitEthernet0/5" ratelimit 60
 action 1.0 syslog priority warnings msg "EEM SELF-HEAL: repeated flap on Gi0/5 - containing"
 action 2.0 cli command "enable"
 action 3.0 cli command "configure terminal"
 action 4.0 cli command "interface GigabitEthernet0/5"
 action 5.0 cli command "shutdown"
 action 6.0 cli command "end"
 action 7.0 syslog priority critical msg "EEM SELF-HEAL: Gi0/5 shut due to flapping - physical layer needs attention"

The philosophy flip here is worth stating explicitly: containment is healing too. An intermittent fault that keeps half-working is often more damaging to the wider network than a clean hard-down. Turning "flapping and destabilizing" into "down and contained" is a legitimate, often superior, self-healing outcome. Just make the announcement scream why the port is down, so nobody spends an hour wondering who shut Gi0/5.

For production you'd typically drive this off a flap count threshold rather than a single UPDOWN (the native errdisable detect cause link-flap with a configured link-flap max-flaps does the counting for you), then use EEM for the announcement and any extra actions. The principle stands regardless of what does the counting.

The self-healing safety checklist

Every self-healing applet you deploy should pass all of these before it touches production. This is the discipline that keeps the whole approach from backfiring:

Rate limit is mandatory, and sized to blast radius. The bigger the action's impact (clearing a routing process ≫ clearing one port), the harder the rate limit. A remediation with no rate limit is a loaded gun.
Capture before you remediate. If the action destroys diagnostic state (CPU spike, adjacency loss), snapshot it first. Healing that erases the evidence is a net loss.
Announce at a severity that matches reality. Routine recovery → notifications. Something that needs eventual human eyes → warnings or critical. The log should let you reconstruct every automated intervention.
Scope narrowly. An applet that matches too broadly (clear ip ospf process on any adjacency change anywhere) is far more dangerous than one scoped to a specific neighbor, interface, or condition. Be as specific as you can.
Prefer the gentlest effective action. Bounce one interface before clearing a process. Soft-clear a BGP neighbor before hard-clearing. Capture-and-escalate before auto-remediating at all.
Never override a security control. If the condition is a security event (port-security, BPDU guard where it matters), self-healing should not quietly undo it.
Test the failure, not just the applet. Actually induce the condition in a lab — pull the cable, spike the CPU, drop the neighbor — and watch the applet do its thing. An untested self-healing applet is a hypothesis about production behavior, and production is a bad place to test hypotheses.

Where this leaves us

Self-healing, done with discipline, quietly removes a whole class of pages from your life — the ones where you were only ever going to run one command anyway. The applets above handle the instance; the announcements make sure you still see the pattern. That combination is the whole art: the network recovers on its own, but never lies to you about the fact that it had to.

The recurring theme you'll have noticed is that the best self-healing action is frequently just "capture everything and escalate" rather than "swing a hammer." That capture-in-the-moment capability deserves its own treatment — because the hardest problems to fix are the ones you can never catch in the act.

Which pages in your rotation are the ones you'd automate away first — the "I just ran one command and went back to bed" kind? Those are the perfect self-healing candidates, and I'm curious which ones top other people's lists.

Visit Website: Digital Security Lab