Beyond Simple Applets: Event Correlation, Tcl, and a Complete Real-World EEM Build

This is the finale of a 5-part series on Cisco EEM. Part 1 built the model; Part 2 healed; Part 3 captured; Part 4 guarded. Everything so far has used single-event applets with linear action lists. This part removes that ceiling — multi-event correlation, environment variables, timer classes, and the drop to Tcl — and then spends its second half on a complete, deployable, two-router production build that ties the whole series together.

---

Every applet in this series so far has followed one shape: a single event, a linear list of actions. That shape covers most of what you'll ever need. But there's a class of problems it can't express — problems that depend on combinations of events, on state carried between runs, on timing relationships, or on logic and string manipulation richer than if/string match. This part is about breaking that ceiling, and then applying everything to a real build.

We'll move fast and stay dense. If you've followed Parts 1–4, you have the vocabulary. Let's use it.

Multi-event correlation: the trigger block

A single applet can register multiple events and fire based on a boolean relationship between them. We touched this in Part 3; here's the full model.

The structure is: declare each event with a tag, then a trigger block with a correlate statement expressing the logic.

event manager applet MULTI-EVENT-DEMO
 event tag e1 syslog pattern "%LINK-3-UPDOWN.*GigabitEthernet0/1.*down"
 event tag e2 track 10 state down
 trigger
  correlate event e1 and event e2
 action 1.0 syslog priority critical msg "EEM CORRELATE: Gi0/1 down AND track 10 down - confirmed path failure"

The correlate expression supports:

• Boolean operators: and, or, and grouping — correlate event e1 and event e2 or event e3.
• Occurrence within the trigger: trigger occurs 3 period 60 — the whole correlated condition must be satisfied 3 times in 60 seconds before the applet fires.
• A trigger period — the window within which the correlated events must all occur to count as correlated.

Why this matters: correlation collapses false positives. In Part 3 we captured on "CPU high AND drops bursting" precisely because either alone is noisy. The same logic applies to remediation — you don't want to act on a single symptom that might be benign; you want to act when a pattern of symptoms confirms a real condition. The trigger/correlate block is how you express "only when several things agree" without leaving applet syntax.

Here's a more operational example: fail over a service only when the primary path is confirmed dead by two independent signals — the interface down AND an IP SLA probe failing — never on just one, which could be a false alarm:

event manager applet CONFIRMED-PATH-DOWN
 event tag link syslog pattern "%LINK-3-UPDOWN.*GigabitEthernet0/1, changed state to down"
 event tag sla syslog pattern "%TRACK-6-STATE.*10 ip sla.*Up -> Down"
 trigger
  correlate event link and event sla
  attribute tag link occurs 1
  attribute tag sla occurs 1
 action 1.0 syslog priority critical msg "EEM: primary path CONFIRMED down by two signals - initiating failover actions"
 action 2.0 cli command "enable"
 action 3.0 cli command "configure terminal"
 action 4.0 cli command "ip route 0.0.0.0 0.0.0.0 198.51.100.9 10"
 action 5.0 cli command "end"
 action 6.0 syslog priority notifications msg "EEM: backup default route installed"

Requiring two signals to agree before touching routing is exactly the kind of discipline that keeps automation from causing the outage it was meant to prevent.

Environment variables: parameterizing applets

Hardcoding values (thresholds, server IPs, interface names) into every applet makes them brittle and hard to maintain across a fleet. EEM supports environment variables — set once, referenced everywhere:

event manager environment _syslog_server 192.168.37.50
event manager environment _cpu_threshold 85
event manager environment _capture_path flash:

Reference them in applets with the $ prefix:

event manager applet CPU-CHECK-PARAM
 event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.6.1 get-type exact entry-op ge entry-val $_cpu_threshold poll-interval 10
 action 1.0 syslog priority warnings msg "EEM: CPU exceeded $_cpu_threshold percent"
 action 2.0 cli command "enable"
 action 3.0 cli command "show processes cpu sorted | append $_capture_path/cpu.txt"

Now the same applet definition works across your fleet, and you tune behavior by changing environment variables rather than editing every applet. For fleet management via templates, this is the difference between maintainable and unmaintainable EEM. Note the convention: user environment variables and EEM's own built-ins both use the _ prefix, so name yours distinctly to avoid colliding with built-ins like $_cli_username.

Timer classes: watchdog, countdown, cron, absolute

Part 1 showed cron timers. EEM has four timer event types, each for a different job:

• cron — calendar schedule ("0 2 * * *"). For "every night at 2 AM."
• watchdog — fires every N seconds continuously. For periodic polling/health checks.
• countdown — fires once after N seconds, then the applet is done. For one-shot delayed actions.
• absolute — fires at one specific date/time. For scheduled one-time events.

A watchdog example — a periodic self-health check that runs every 5 minutes and validates critical state, escalating if something's wrong:

event manager applet HEALTH-WATCHDOG
 event timer watchdog time 300
 action 1.0 cli command "enable"
 action 2.0 cli command "show ip interface brief | include down"
 action 3.0 string match "*down*" "$_cli_result"
 action 4.0 if $_string_result eq "1"
 action 5.0  syslog priority warnings msg "EEM WATCHDOG: one or more interfaces down - review"
 action 6.0 end

The watchdog timer is the engine for any "check this every N seconds and react" pattern — a lightweight, on-box alternative to external polling for conditions you want evaluated continuously and locally.

Applet-level controls you should always consider

Before we leave applet syntax, four qualifiers that belong in your muscle memory:

• ratelimit <sec> — minimum seconds between firings. (Hammered on throughout this series — never omit it on anything that can fire rapidly.)
• maxrun <sec> — maximum runtime before EEM kills the applet. Raise it for applets running many show commands or slow operations (Part 3's post-reload capture needed this).
• nice 1 — run the applet at lower CPU priority, so a heavy applet doesn't compete with the control plane during an already-stressed moment.
• authorization bypass and sync {yes|no} — control whether the applet runs synchronously (able to gate a command) or asynchronously, and how it interacts with AAA command authorization.

These aren't decorative. On a busy device, an applet without maxrun and nice can itself become a problem during the very incident it's responding to.

When to drop to Tcl

Applets are declarative and readable, but they hit walls:

• Complex string parsing — extracting and transforming multiple fields from command output.
• Loops and data structures — iterating over a list of interfaces, building arrays, accumulating state.
• Arithmetic beyond trivial — real calculations, not just threshold comparisons.
• Multi-step logic with branching that if/else makes unreadable.
• Reusable procedures shared across triggers.

For these, EEM runs full Tcl scripts. The event registration happens inside the script via ::cisco::eem::event_register_*, and you get the entire Tcl language plus Cisco's EEM library. Here's the shape of a Tcl policy that registers on syslog and parses output — the skeleton you'd extend:

::cisco::eem::event_register_syslog pattern "%HSRP-5-STATECHANGE"

namespace import ::cisco::eem::*
namespace import ::cisco::lib::*

# Query the event that triggered us
array set arr_einfo [event_reqinfo]
set msg $arr_einfo(msg)

# Parse the interface and new state out of the message
if {[regexp {(\S+) Grp (\d+) state (\S+) -> (\S+)} $msg -> intf grp oldstate newstate]} {
    # Open a CLI channel
    if {[catch {cli_open} result]} {
        error $result $errorInfo
    }
    array set cli $result

    cli_exec $cli(fd) "enable"

    # Branch on the parsed new state
    if {$newstate == "Active"} {
        cli_exec $cli(fd) "configure terminal"
        cli_exec $cli(fd) "interface GigabitEthernet0/2"
        cli_exec $cli(fd) "no shutdown"
        cli_exec $cli(fd) "end"
        action_syslog priority notifications msg "EEM Tcl: became Active on Grp $grp - Gi0/2 UP"
    } elseif {$oldstate == "Active"} {
        cli_exec $cli(fd) "configure terminal"
        cli_exec $cli(fd) "interface GigabitEthernet0/2"
        cli_exec $cli(fd) "shutdown"
        cli_exec $cli(fd) "end"
        action_syslog priority notifications msg "EEM Tcl: left Active on Grp $grp - Gi0/2 DOWN"
    }

    cli_close $cli(fd) $cli(tty_id)
}

Register a Tcl policy with:

event manager directory user policy flash:/eem-policies
event manager policy hsrp-track.tcl type user

The power here is the regexp that extracts intf, grp, oldstate, and newstate as variables in one line, then branches on them — logic that's clumsy in applet syntax and natural in Tcl. For anything involving real parsing or iteration, Tcl is the right tool. For straightforward trigger-and-act, applets remain clearer and easier to audit. Choose deliberately: readability and auditability are worth a lot in a production network, so don't reach for Tcl until an applet genuinely can't express the logic.

---

The Complete Build: Automating an HSRP-Ineligible Interface Across Two Routers

Now we put it together. This is the capstone: a full, deployable, two-router branch build where an HSRP-ineligible interface is kept in lockstep with the HSRP active/standby role by EEM. It uses the correlation-free, applet-based approach (because it's cleaner and more auditable here), and it embeds the single most important practical lesson of the entire series — the HSRP state-transition regex almost everyone gets wrong.

The problem, precisely

You have a branch router pair running HSRP for the LAN gateway. There's one interface — call it the service interface — that cannot participate in HSRP: it's a point-to-point /30 handoff to a provider, structurally ineligible for a first-hop redundancy protocol. But operationally it must be up on exactly the router that is currently HSRP active, and down on the standby, so the provider only ever sees one live handoff and it's always on the active router.

HSRP knows which router is active. The service interface has no native way to follow. EEM is the bridge.

Topology and addressing

        LAN 192.168.10.0/24         VIP: 192.168.10.1  (HSRP grp 1)
        ┌─────────────────────────────────────────────┐
        │                                               │
   ┌────┴─────┐                                   ┌─────┴────┐
   │    R1    │  Gi0/0 .2   HSRP pri 110          │    R2    │  Gi0/0 .3   HSRP pri 100
   │ (ACTIVE) │  Gi0/1  WAN 203.0.113.2/30        │(STANDBY) │  Gi0/1  WAN 203.0.113.6/30
   │          │  Gi0/2  SERVICE 198.51.100.1/30   │          │  Gi0/2  SERVICE 198.51.100.1/30
   └──────────┘  (HSRP-ineligible, EEM-managed)   └──────────┘  (HSRP-ineligible, EEM-managed)

• R1 — HSRP priority 110, preempt: the intended active.
• R2 — HSRP priority 100: the standby.
• Gi0/2 — the service interface, same /30 on both, shutdown at baseline on both, brought up by EEM only on whichever router is HSRP active.

The shutdown baseline is a load-bearing design decision: neither router drives the interface until EEM says so, which guarantees no both-active window at boot before HSRP settles.

Part A — Shared baseline (both routers)

Identical except identity fields. Abbreviated here to the parts that matter for this build; use your full hardening baseline in production.

! ===== BASELINE (R1 shown; R2 identical bar hostname/mgmt IP) =====
hostname R1-BRANCH
!
service timestamps log datetime msec localtime show-timezone
service timestamps debug datetime msec localtime show-timezone
service password-encryption
!
! Logging is MANDATORY - EEM syslog triggers read the buffer
logging buffered 65536 informational
logging origin-id hostname
logging host 192.168.37.50
!
! NTP - both peers MUST share time for coherent failover logs
ntp server 192.168.37.10 prefer
ntp server 192.168.37.20
!
ip domain name branch.example.com
ip ssh version 2
!
username localadmin privilege 15 secret <STRONG_SECRET>
enable secret <STRONG_ENABLE_SECRET>
!
line con 0
 logging synchronous
 exec-timeout 10 0
 login local
line vty 0 4
 transport input ssh
 login local
 exec-timeout 10 0
!
end

The two lines that are non-negotiable for this build:

• logging buffered ... informational — the EEM event syslog triggers read the logging buffer. HSRP %HSRP-5-STATECHANGE messages are severity 5. If the buffer level is above 5 or logging is off, the applets silently never fire. This is the number-one reason an HSRP-EEM build "doesn't work."
• service timestamps log datetime msec — millisecond timestamps let you correlate, across both routers, exactly when HSRP flipped and when each applet reacted. Indispensable during failover validation.

Part B — Interfaces and HSRP

R1 (active)

! ===== R1 INTERFACES + HSRP =====
!
interface GigabitEthernet0/0
 description LAN-SEGMENT
 ip address 192.168.10.2 255.255.255.0
 standby version 2
 standby 1 ip 192.168.10.1
 standby 1 priority 110
 standby 1 preempt delay minimum 30
 standby 1 timers msec 250 msec 750
 standby 1 track 10 decrement 30
 no shutdown
!
interface GigabitEthernet0/1
 description WAN-UPLINK
 ip address 203.0.113.2 255.255.255.252
 no shutdown
!
! Service interface - HSRP-INELIGIBLE - EEM-managed - shutdown at baseline
interface GigabitEthernet0/2
 description SERVICE-HANDOFF-HSRP-INELIGIBLE
 ip address 198.51.100.1 255.255.255.252
 shutdown
!
! Track the WAN uplink via IP SLA so a usable-path failure (not just link down)
! drives HSRP failover
ip sla 10
 icmp-echo 203.0.113.1 source-interface GigabitEthernet0/1
 frequency 5
ip sla schedule 10 life forever start-time now
!
track 10 ip sla 10 reachability

R2 (standby)

! ===== R2 INTERFACES + HSRP =====
!
interface GigabitEthernet0/0
 description LAN-SEGMENT
 ip address 192.168.10.3 255.255.255.0
 standby version 2
 standby 1 ip 192.168.10.1
 standby 1 priority 100
 standby 1 preempt delay minimum 30
 standby 1 timers msec 250 msec 750
 standby 1 track 10 decrement 30
 no shutdown
!
interface GigabitEthernet0/1
 description WAN-UPLINK
 ip address 203.0.113.6 255.255.255.252
 no shutdown
!
interface GigabitEthernet0/2
 description SERVICE-HANDOFF-HSRP-INELIGIBLE
 ip address 198.51.100.1 255.255.255.252
 shutdown
!
ip sla 10
 icmp-echo 203.0.113.5 source-interface GigabitEthernet0/1
 frequency 5
ip sla schedule 10 life forever start-time now
!
track 10 ip sla 10 reachability

Note the symmetry: both routers are configured identically for the service interface (same address, same shutdown baseline). The only thing that determines which router drives Gi0/2 is which one is HSRP active — and that's decided by priority, preempt, and tracking. EEM simply translates "am I active?" into "is Gi0/2 up?"

Part C — The critical piece: the HSRP state-transition regex

This is the single most important technical point in the entire series, so we'll be exact.

When HSRP changes state, IOS logs %HSRP-5-STATECHANGE. The naive assumption is that failover is a clean Standby -> Active or Active -> Standby. It is not. HSRP runs a state machine — Init, Listen, Speak, Standby, Active — and depending on why the change happened, the router passes through different intermediate states. Real transitions you will see include:

%HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Standby -> Active
%HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Listen -> Active
%HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Speak -> Active
%HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Active -> Speak
%HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Active -> Standby
%HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Active -> Init

If your applet matches only Standby -> Active, it will miss a failover that arrives via Listen -> Active or Speak -> Active — which happens routinely depending on timing and cause. The interface then fails to come up, and you have a broken automation that works sometimes, which is worse than one that never works because it's harder to catch.

The fix: don't enumerate transitions. Anchor on the presence of Active on the correct side of the arrow.

• To catch becoming active — any transition into Active — match: Grp 1 state .* -> Active
• To catch leaving active — any transition out of Active — match: Grp 1 state Active ->

No matter which intermediate state HSRP traverses, if it ends in Active you catch it, and if it leaves Active you catch it. This is the difference between a build that works every time and one that works most of the time.

Part D — The EEM applets (identical on both routers)

Because the applets key off the router's own HSRP state, they're identical on R1 and R2. Each router independently drives its own Gi0/2 based on its own role. Combined with the shutdown baseline, whichever router is active ends up owning the interface.

! ===== EEM APPLETS - deploy IDENTICALLY on R1 and R2 =====
!
! Became HSRP active (via ANY intermediate state) -> bring service interface UP
event manager applet HSRP-BECAME-ACTIVE
 event syslog pattern "%HSRP-5-STATECHANGE.*Grp 1 state .* -> Active" ratelimit 10
 action 1.0 syslog priority notifications msg "EEM HSRP: transitioned INTO Active - bringing Gi0/2 up"
 action 2.0 cli command "enable"
 action 3.0 cli command "configure terminal"
 action 4.0 cli command "interface GigabitEthernet0/2"
 action 5.0 cli command "no shutdown"
 action 6.0 cli command "end"
 action 7.0 syslog priority notifications msg "EEM HSRP: Gi0/2 service interface is now UP (this router is HSRP active)"
!
! Left HSRP active (via ANY intermediate state) -> shut service interface DOWN
event manager applet HSRP-LEFT-ACTIVE
 event syslog pattern "%HSRP-5-STATECHANGE.*Grp 1 state Active ->" ratelimit 10
 action 1.0 syslog priority notifications msg "EEM HSRP: transitioned OUT OF Active - shutting Gi0/2 down"
 action 2.0 cli command "enable"
 action 3.0 cli command "configure terminal"
 action 4.0 cli command "interface GigabitEthernet0/2"
 action 5.0 cli command "shutdown"
 action 6.0 cli command "end"
 action 7.0 syslog priority notifications msg "EEM HSRP: Gi0/2 service interface is now DOWN (this router is not HSRP active)"

Design points:

• ratelimit 10 — protects the service interface from thrashing if HSRP flaps. Combined with the preempt delay minimum 30 on the HSRP config, a brief instability won't bounce Gi0/2.
• Breadcrumbs on every path — the notifications-severity messages record every transition and action, on both routers, so show logging | include EEM HSRP reconstructs the entire failover story.
• Stateless and symmetric — the applets don't know or care whether they're on R1 or R2. They react to local HSRP state. That's what makes them deployable identically.

Part E — Lifecycle walk-through

Trace the states to prove the design:

Boot. Both routers boot with Gi0/2 shut. HSRP elects: R1 (pri 110, preempt) → Active; R2 → Standby. R1's HSRP-BECAME-ACTIVE fires on the -> Active transition, brings R1's Gi0/2 up. R2 never entered Active, so R2's Gi0/2 stays down. Result: service up on R1 only. ✓

R1 uplink fails. R1's IP SLA 10 fails → track 10 down → HSRP priority 110−30=80, below R2's 100. R1 transitions out of Active (e.g. Active -> Speak); HSRP-LEFT-ACTIVE fires, shuts R1's Gi0/2. R2 transitions into Active (e.g. Standby -> Active, possibly via Listen/Speak); HSRP-BECAME-ACTIVE fires, brings R2's Gi0/2 up. Result: service moved to R2. ✓ — and note this works because the regex catches whatever intermediate path each router took.

R1 recovers. IP SLA 10 succeeds → track 10 up → R1 priority restores to 110 → after preempt delay minimum 30, R1 retakes Active. R1's applet brings Gi0/2 up; R2 leaves Active and its applet shuts R2's Gi0/2. Result: service back on R1. ✓ The preempt delay prevents a flapping uplink from bouncing the service interface repeatedly.

Part F — Verification and testing

Confirm applets are registered on both routers:

show event manager policy registered

Confirm HSRP roles:

! R1: Active, priority 110    R2: Standby, priority 100
show standby brief

Confirm the tracking:

show track 10
show ip sla statistics 10

Now force a failover in a maintenance window and watch both routers simultaneously. Shut R1's uplink (or test the SLA down), and observe:

! Confirm each applet fired and on which event:
show event manager history events
!
! Watch the EEM breadcrumbs:
show logging | include EEM HSRP
!
! Confirm HSRP role changed:
show standby brief
!
! Confirm service interface state - must be UP on exactly one router:
show ip interface brief | include GigabitEthernet0/2

The acceptance criterion: after failover, Gi0/2 is up on exactly one router (the new active) and down on the other. If HSRP changed role but no EEM breadcrumb appeared, the regex didn't match the transition — verify you used the -> Active / Active -> anchoring and not enumerated transitions.

Part G — Persist and harden

Save on both routers — the shutdown baseline must be in startup-config so boot behavior is correct after a power event:

copy running-config startup-config

And a final guardrail, tying back to Part 4: alert if anyone modifies the EEM config itself, so the automation can't be silently disabled:

event manager applet GUARD-EEM-CHANGE
 event syslog pattern "%HA_EM-6-LOG.*event manager" ratelimit 30
 action 1.0 syslog priority critical msg "EEM GUARD: EEM configuration was modified - verify HSRP automation intact"

When to use this pattern — and when to fix the design instead

This build is the right answer when an interface structurally cannot run HSRP/VRRP but genuinely must follow the active role. That's a real and common situation, and EEM handles it cleanly.

It is the wrong answer when the interface could be made eligible. Native first-hop redundancy is always more robust than syslog-triggered automation — it has no dependency on logging, no regex to get wrong, no applet to be removed. If you can solve it with standby commands, do that. Reach for this pattern only when native redundancy genuinely isn't available for that interface.

The series, in one idea

Five parts, one thread: EEM is the automation that lives on the device and keeps working when everything else can't reach it. We used it to heal, to capture the un-catchable, to guard against mistakes and drift, and finally to bridge a gap native features couldn't — always on the box, always in the moment, always leaving a trail.

The primitives never changed: an event, some actions, a syslog breadcrumb, a rate limit, a threshold. What changed was ambition. Start with one applet that recovers a port. End with a two-router build that keeps a provider handoff pinned to the active gateway through every failover path HSRP can take. Same tool. The distance between those two is just practice — and now you have the map.

Thanks for reading the series. Go build something that fixes itself.

---

This wraps the EEM series. If you deploy any of these — especially the HSRP build — I'd genuinely like to hear how it went, what you adapted, and what broke. The -> Active regex lesson cost me real troubleshooting time to learn; if this saved you some, that's the whole point.

Visit Website: Digital Security Lab