On February 3, 2020, the Iowa Democratic Party attempted to report precinct caucus results using a new mobile reporting application that had been deployed with inadequate testing, inadequate training, and no meaningful load testing against the expected reporting volume. Within hours of caucus completion, the app had failed to report results correctly for hundreds of precincts. The fallback — a telephone reporting system — was overwhelmed. Results were delayed by days. The incident did not change any election outcome; no votes were altered. But the operational failure created a public confidence crisis that persisted independently of the factual outcome, and that caused measurable and lasting damage to the institution conducting the election.
The Iowa caucus application failure is the most visible recent example of a class of election infrastructure failure that has been recurring in various forms for two decades: a new system deployed without the reliability engineering rigour that the stakes of the deployment require. The systems managing voter registration, ballot processing, and results reporting are not commercial applications. They operate under non-renewable deadlines, adversarial threat conditions, and a public verifiability requirement that commercial SLA frameworks do not contemplate. The consequence of failure is not revenue loss or customer churn — it is degraded public trust in democratic institutions.
Site Reliability Engineering does not solve the political dimensions of election administration. It does solve the engineering dimensions — and the engineering dimensions are substantial, well-defined, and addressable with the same principles that protect commercial critical infrastructure.
What Makes Election Infrastructure Uniquely Challenging
Election systems share reliability challenges with other critical infrastructure — availability requirements, adversarial threat exposure, regulatory oversight. Three characteristics distinguish them from every other system class addressed in this series.
The Non-Renewable Deadline
In financial services, a missed settlement window can be corrected the next business day. In healthcare, a system unavailability event triggers downtime procedures that degrade but do not eliminate care delivery. In elections, Election Day is a non-renewable deadline. If the voter registration lookup system is unavailable at 7:00 AM when polls open, there is no "next window." The failure must be resolved within the hours the polls are open, or provisional ballot procedures are the fallback — and provisional ballot reconciliation is itself an error-prone, resource-intensive process.
This non-renewable deadline constraint transforms the standard reliability engineering calculus. MTTR targets that are acceptable in commercial contexts — 30-minute, one-hour — are not acceptable for the Election Day operational window. For the systems that are critical during voting hours, MTTR must be measured in minutes, and the primary engineering investment is in prevention and fast detection, not remediation.
The Adversarial Threat Dimension
Commercial critical infrastructure faces adversarial threats (ransomware, DDoS). Election infrastructure faces adversarial threats with a specific additional characteristic: the adversary's goal may be to create the appearance of unreliability without actually compromising results — to create public doubt about election integrity rather than to actually alter election outcomes.
This means that election infrastructure SRE frameworks must address two distinct failure modes simultaneously: genuine operational failures that prevent accurate vote counting, and adversarial interference that creates reputational or confidence damage without altering outcomes. The audit trail and public verifiability SLIs addressed later in this post are the engineering response to the second failure mode.
The Public Verifiability Requirement
Commercial systems are accountable to their SLAs and their regulators. Election systems are accountable to the public. The audit trail is not a compliance artefact — it is the evidence base that enables public confidence in the outcome. A system that correctly counts every vote but cannot demonstrate that it did so to an independently verifiable standard has failed at one of its core requirements, regardless of whether the vote count was accurate.
This means that for election systems, the observability architecture is not just an operational tool. It is a public accountability instrument. The logs, the configuration records, the deployment history, and the operational event timeline are evidence in a potential post-election audit. They must be designed with that use case in mind.
SLI Design for Election Infrastructure
Election infrastructure requires six SLI dimensions. Four correspond to the Four Golden Signals adapted to the electoral context. Two are election-specific: Integrity and Auditability.
────────────────────────────────────────────────────────────────────────────
SIX SLI DIMENSIONS FOR ELECTION INFRASTRUCTURE
DIMENSION 1: AVAILABILITY
Voter registration lookup:
SLI: Fraction of voter lookup requests returning valid response
within 3 seconds during active polling hours
Target: 99.99% during polling window (non-renewable deadline)
Target: 99.9% during non-polling periods
Results reporting system:
SLI: Fraction of precinct result submission requests
accepted and confirmed within 30 seconds
Target: 99.95% during results reporting window
DIMENSION 2: LATENCY
Poll worker interface: p95 response < 2 seconds
(poll worker queue forms with voters waiting)
Ballot processing throughput: N ballots per hour
(must exceed peak county ballot processing rate)
Results reporting: precinct result confirmation < 30 seconds
DIMENSION 3: ERRORS
Voter registration database: failed lookup rate
(high failure rate may indicate data integrity issues)
Ballot processing rejection rate: ballots rejected by scanner
(anomalous rejection rate may indicate equipment failure
or ballot stock issues)
Results transmission failures: precincts failing to report
(tracking, not just counting — each precinct must be accounted for)
DIMENSION 4: SATURATION
Registration database connection pool during Election Day peak
Results reporting API capacity at simultaneous precinct reporting
Network bandwidth to county aggregation systems
DIMENSION 5: INTEGRITY (Election-specific)
SLI: Fraction of ballots for which chain-of-custody records
are complete and consistent across all system records
Target: 100.000% — every ballot must be accountable
Source: Automated consistency check across ballot tracking,
scanner log, and tabulation system records
SLI: Fraction of voter registration records where data matches
across all authoritative sources (state voter file,
county records, precinct assignments)
Target: 99.99% — inconsistencies create provisional ballot burden
Source: Automated cross-system reconciliation
DIMENSION 6: AUDITABILITY (Election-specific)
SLI: Fraction of system events (logins, configuration changes,
data modifications) that are captured in the tamper-evident
audit log within 60 seconds of occurrence
Target: 100.000% — no unaudited events during election period
Source: Audit log completeness check against event counters
SLI: Fraction of audit log entries that are verifiably unmodified
(cryptographic hash check on audit record integrity)
Target: 100.000%
────────────────────────────────────────────────────────────────────────────
The Non-Renewable Deadline SLO Architecture
The standard SLO measurement window — 28 days rolling — is not meaningful for election systems that operate in distinct operational phases with radically different reliability requirements. Election infrastructure SLOs must be defined against operational phases, not calendar windows.
────────────────────────────────────────────────────────────────────────────
ELECTION INFRASTRUCTURE OPERATIONAL PHASES AND SLO TARGETS
PHASE 1 — VOTER REGISTRATION PERIOD (30–90 days before Election Day)
Systems active: Voter registration database, online registration portal
Traffic pattern: moderate steady load with deadline spikes
Availability SLO: 99.9% (< 8.8 hours downtime during period)
Latency SLO: p95 < 3 seconds for registration submissions
Integrity SLO: 100% record consistency within 24 hours of submission
Deployment policy: standard; changes permitted with normal gates
PHASE 2 — FINAL REGISTRATION CLOSE (7 days before Election Day)
Systems active: All registration systems + poll book generation
Traffic pattern: high deadline-driven spike
Availability SLO: 99.99% (< 52 minutes downtime during period)
Deployment policy: FROZEN — no changes to voter registration systems
Exception: security patches with CISA advisory require joint
approval: Election Director + State IT + CISA coordination
PHASE 3 — ELECTION DAY (Polling window: typically 12–15 hours)
Systems active: Voter lookup, poll book access, provisional ballot
tracking, accessibility accommodation systems
Traffic pattern: high sustained with morning and evening peaks
Availability SLO: 99.999% during polling hours
(< 5 minutes outage during polling window)
MTTR SLO: < 15 minutes for any degradation
Deployment policy: ABSOLUTE FREEZE — no changes under any circumstances
during polling hours
Incident response: pre-positioned county IT staff; hot standby systems;
offline poll book printouts as Tier 0 fallback
PHASE 4 — RESULTS REPORTING (Evening of Election Day + canvass period)
Systems active: Results submission, aggregation, publication
Traffic pattern: high burst at close of polls across all precincts
Availability SLO: 99.95% during initial results reporting window
Integrity SLO: 100% — every reported result matches source precinct
Deployment policy: FROZEN during active reporting
Audit requirement: every results submission logged with submitter
identity, timestamp, precinct ID, and cryptographic signature
PHASE 5 — CANVASS AND CERTIFICATION (Days to weeks post-election)
Systems active: Audit tools, recount support, certification workflows
Availability SLO: 99.9% (standard business hours)
Auditability SLO: 100% — every record must be producible on demand
Deployment policy: standard; changes require audit trail annotation
────────────────────────────────────────────────────────────────────────────
Chaos Engineering for Election Systems
Chaos engineering — the deliberate injection of failure conditions to test system resilience — is standard SRE practice. For election systems, it requires specific constraints that commercial system chaos engineering does not.
────────────────────────────────────────────────────────────────────────────
ELECTION INFRASTRUCTURE CHAOS ENGINEERING PROTOCOL
WHEN TO RUN: Only during Phases 1 and outside election years for
primary systems. Never during Phases 2–5.
Dedicated non-production election test environment required.
Production chaos testing: PROHIBITED for election systems.
SCENARIO LIBRARY FOR ELECTION INFRASTRUCTURE:
Scenario 1: Registration Database Unavailability
Inject: Database primary node failure
Measure: Failover time, data consistency post-failover
Pass criteria: Failover < 30 seconds; zero data loss; zero corruption
Frequency: Quarterly
Scenario 2: Election Day Peak Load Surge
Inject: 3× expected peak load during morning poll opening simulation
Measure: Latency degradation, error rate, autoscaling response
Pass criteria: p95 latency < 3s; error rate < 0.01%; scale response < 90s
Frequency: 60 days before each major election
Scenario 3: Results Reporting Cascade
Inject: Simultaneous result submissions from all precincts
Measure: System throughput, queue depth, data integrity under load
Pass criteria: 100% of submissions accepted within 60s; 100% integrity
Frequency: 60 days before each major election
Scenario 4: Network Partition at County Aggregation
Inject: Network partition between precinct reporting system
and county aggregation system
Measure: Graceful degradation; data buffering; recovery completeness
Pass criteria: Zero data loss; automatic recovery on reconnection
Frequency: Annually
Scenario 5: Audit Log Integrity Under Attack
Inject: Simulated attempt to modify audit log records
Measure: Tamper detection time; alerting accuracy
Pass criteria: Tamper detected within 60 seconds; alert fires
Frequency: Quarterly
────────────────────────────────────────────────────────────────────────────
ADVERSARIAL SCENARIO LIBRARY (Red Team, not standard chaos):
These scenarios require security team involvement and may require
coordination with CISA's election security advisors.
→ Credential stuffing against election worker authentication systems
→ DDoS against voter lookup APIs at Election Day volumes
→ Supply chain attack simulation against election software updates
→ Social engineering targeting IT staff with administrative access
────────────────────────────────────────────────────────────────────────────
Auditability Architecture: GitOps as Electoral Chain of Custody
The GitOps operational model — where every configuration change is a git commit, every deployment is traceable to a specific commit, and every drift from desired state is detected and logged — is a natural fit for the chain-of-custody requirements of election infrastructure. The properties that make GitOps valuable for operational governance (tamper-evident change history, automated drift detection, declarative desired state) are the same properties that satisfy post-election audit requirements.
# Argo CD Application — Election Voter Registration System
# GitOps provides chain-of-custody for all configuration changes
# Every sync event is a time-stamped, identity-attributed change record
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: voter-registration-system
namespace: argocd
annotations:
# All sync events → Splunk tamper-evident audit log
notifications.argoproj.io/subscribe.on-sync-succeeded.splunk: "election-audit"
notifications.argoproj.io/subscribe.on-sync-failed.splunk: "election-audit"
notifications.argoproj.io/subscribe.on-health-degraded.splunk: "election-audit"
# Phase-based deployment gate annotation
election.internal/operational-phase: "1" # Updated as phases progress
election.internal/deployment-freeze: "false" # Set to "true" in Phase 2+
spec:
project: election-infrastructure
source:
repoURL: https://git.internal/election/infrastructure
targetRevision: main
path: clusters/election/voter-registration
destination:
server: https://tkg-election.internal:6443
namespace: election-systems
syncPolicy:
automated:
prune: true
selfHeal: true # Drift auto-remediated; every remediation is audit-logged
syncOptions:
- ServerSideApply=true
# Kyverno Policy — Phase-Based Deployment Gate
# Enforces election operational phase restrictions at admission time
# Phase annotation on Application resource controls what is permitted
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: election-phase-deployment-gate
annotations:
policies.kyverno.io/description: >
Enforces election operational phase deployment restrictions.
Phases 2–5 have increasingly restrictive deployment policies.
Phase annotations on Application resources are the authoritative
state; this policy enforces them automatically.
spec:
validationFailureAction: Enforce
rules:
- name: block-deployment-during-freeze
match:
any:
- resources:
kinds: [Application]
namespaces: [argocd]
selector:
matchLabels:
election.internal/system: "true"
preconditions:
all:
- key: "{{ request.object.metadata.annotations.\"election.internal/deployment-freeze\" }}"
operator: Equals
value: "true"
validate:
message: >
Deployment blocked: election system is in operational phase with
deployment freeze active. Changes require Election Director +
State IT joint approval with CISA coordination for security patches.
deny:
conditions:
all:
- key: "{{ request.object.metadata.annotations.\"election.internal/freeze-override-approved\" }}"
operator: NotEquals
value: "true"
CISA Election Infrastructure Security Framework Alignment
CISA designates election infrastructure as critical infrastructure and provides security guidance that maps directly to SRE operational practices.
────────────────────────────────────────────────────────────────────────────
CISA ELECTION SECURITY GUIDANCE → SRE MAPPING
CISA RECOMMENDATION: Implement and test backup systems and procedures
SRE MAPPING:
→ Hot standby voter registration database (failover < 30 seconds)
→ Offline poll book printouts as Tier 0 fallback (generated 24h before)
→ Quarterly failover tests documented as chaos engineering exercises
→ Results reporting: paper precinct tally sheets as authoritative backup
CISA RECOMMENDATION: Maintain comprehensive logs of all system activity
SRE MAPPING:
→ Splunk Enterprise: all authentication, configuration changes,
data modifications logged with structured fields
→ GitOps audit trail: all infrastructure changes as git commits
→ Tamper-evident audit log: cryptographic hash chain on log records
→ Auditability SLI: 100% of events captured within 60 seconds
CISA RECOMMENDATION: Conduct post-election audits of equipment and software
SRE MAPPING:
→ Post-election evidence synthesis automation (Class 4)
→ Splunk query producing complete change log for election period
→ Argo CD sync history as configuration chain-of-custody
→ Results submission audit trail with cryptographic signatures
CISA RECOMMENDATION: Test incident response plans before Election Day
SRE MAPPING:
→ Chaos engineering scenario library (run 60 days before election)
→ Tabletop exercises with county IT, state IT, and CISA advisors
→ MTTR measurement against Election Day operational SLO targets
→ Pre-positioned incident response resources at county level
CISA RECOMMENDATION: Implement multifactor authentication
SRE MAPPING:
→ Kyverno policy enforcing MFA annotation on all election system
service accounts and human access paths
→ Istio STRICT mTLS for all inter-service communication
→ Access review automation: quarterly review with automated evidence
────────────────────────────────────────────────────────────────────────────
The 2-Year Calibration Problem
One of the most difficult SRE challenges unique to election infrastructure is the calibration problem: meaningful elections occur every two to four years, which means that SLO validation cycles, load test calibration, and operational experience accumulation all operate on a dramatically slower cadence than commercial systems.
────────────────────────────────────────────────────────────────────────────
THE CALIBRATION PROBLEM: SOLUTIONS
PROBLEM: You cannot derive SOT for an Election Day load pattern from
production data more frequently than once per election cycle.
Error budget policy targets set from one election's data may
not be valid for the next election's different voter turnout,
different system version, or different threat landscape.
SOLUTION 1: Synthetic Election Day Load Testing
Generate synthetic Election Day load profiles from:
→ Registration database size (known)
→ Historical turnout rates adjusted for current registration
→ Precinct count and submission timing model (known from statute)
Run SOT derivation load tests using synthetic profile annually,
not only in election years. Maintains calibration between elections.
SOLUTION 2: Primary/General Election Calibration Cascade
Primary elections (lower turnout) are calibration runs for general
elections (higher turnout). Use primary election operational data
to validate and adjust SLO targets, SOT values, and chaos engineering
scenarios before the higher-stakes general election.
SOLUTION 3: Cross-Jurisdiction Learning Consortium
Election infrastructure reliability learnings are not competitive
intelligence. States and counties that share operational data,
incident reports, and load test results collectively improve the
calibration baseline available to all participants.
CISA's election security information sharing infrastructure (ISAC)
is the existing mechanism; SRE operational data belongs in it.
PROBLEM: SRE staff turnover between elections means operational
knowledge is lost between election cycles.
SOLUTION: Operational Postmortem + Runbook Corpus
Every election cycle produces an operational postmortem documenting:
→ Actual vs. predicted load (calibration data for next cycle)
→ System failures and resolutions (runbook updates)
→ Near-misses and their detection (alerting improvements)
→ Manual interventions (automation candidates)
This corpus is the institutional memory that survives staff turnover.
────────────────────────────────────────────────────────────────────────────
Common Antipatterns
The New Application Before Major Election antipattern → Deploying a new results reporting application or voter registration system for the first time in a major election without production load testing at election-day volumes, without parallel operation with the previous system for a full election cycle, and without a tested rollback to the previous system if the new one fails. The Iowa caucus application is the archetypal example. New election systems should be first deployed in low-stakes elections (primaries, off-cycle local elections) at least one full cycle before major general elections.
The Paper Backup Complacency antipattern → Treating the existence of paper ballot backups as a complete answer to the election infrastructure reliability requirement. Paper ballots are the ultimate integrity backstop — they provide the authoritative record from which electronic tabulation can be verified. They are not a substitute for reliable electronic systems during the voting period. Poll workers conducting a manual check-in process because the electronic poll book is unavailable process voters at approximately one-third the rate of the electronic system, creating lines that suppress turnout.
The Security-as-Isolation antipattern → Addressing the adversarial threat dimension of election infrastructure exclusively through network isolation and air-gapping, without the observability architecture needed to detect anomalous behaviour within the isolated network. Air-gapped systems that are not monitored have been compromised. The choice is not between observability and security; it is between observed security and unobserved security.
The One-Time Chaos Testing antipattern → Running chaos engineering scenarios once as part of a pre-election certification process and not repeating them. System changes, staff turnover, and infrastructure evolution between elections change the failure mode landscape. Chaos scenarios that passed two years ago do not validate that the current system, with its current configuration and current staff, will respond the same way.
The Audit Log Afterthought antipattern → Designing the election system and treating the audit log as a post-hoc addition rather than a first-class architectural requirement. Audit logs that are designed after the system are invariably incomplete — they capture the events the designers thought to add logging for, not the events that a post-election audit will actually need. Auditability SLI design must happen at system design time, not at deployment time.
Maturity Progression
────────────────────────────────────────────────────────────────────────────
STAGE ELECTION INFRA RELIABILITY NORTH STAR SIGNAL
────────────────────────────────────────────────────────────────────────────
Reactive No phase-based SLOs. Chaos New systems deployed
testing absent. Audit log for first time in
is compliance afterthought. major elections.
Paper fallback is the Chaos scenarios
reliability strategy. untested.
Defined Phase-based SLO architecture Phase deployment
documented. Six SLI freeze policy
dimensions instrumented. implemented. Audit
Chaos scenario library completeness SLI
defined. instrumented.
Measured Chaos engineering run Election Day MTTR
60 days before election. measured. Post-election
SOT derived from synthetic postmortem corpus
load profile. CISA guidance established. Cross-
mapped to SRE practices. jurisdiction data
shared.
Optimised New systems piloted in No Iowa-class failures.
primaries before generals. MTTR < 15 minutes
GitOps chain-of-custody during polling window.
satisfies post-election Audit log completeness
audit automatically. 100% maintained.
Generative SRE framework adopted by CISA references
state election authorities. framework in election
Cross-jurisdiction ISAC security guidance.
data includes SRE Post-election audits
operational telemetry. use SRE audit trail
as primary evidence.
────────────────────────────────────────────────────────────────────────────
Five Action Items for This Week
Define the operational phases for your election infrastructure and document the SLO target for each phase. The phase-based SLO architecture is the most important structural change that election infrastructure SRE enables. Even if instrumentation is not yet in place, the phase definitions and targets create the policy framework that governs deployment decisions, change freeze windows, and incident response priorities.
Audit your current audit log against the Auditability SLI definition. For every system event type (login, configuration change, data modification, result submission), verify that the event is captured in a tamper-evident log within 60 seconds of occurrence. The gaps you find are the evidence base vulnerabilities that a post-election challenge would exploit.
Build the synthetic Election Day load profile and run it against your voter registration system. Use your current voter registration count, historical turnout rates, and polling hours to derive the expected peak lookup rate. Run that load profile against your system — not just the average load but the morning-open and afternoon peak patterns. The gap between what your system can handle and what Election Day requires is your most important reliability risk metric.
Document your Tier 0 fallback for each election-critical system. For the voter registration lookup system: what is the offline poll book generation process, how current is the offline poll book at any given time, and how does poll worker check-in throughput compare between electronic and offline modes? The answers to these questions determine whether your fallback actually maintains election operations or creates a throughput bottleneck that suppresses turnout.
Schedule your pre-election chaos engineering scenarios on the calendar now, 60+ days before the next election. The most commonly skipped step in election infrastructure reliability preparation is chaos engineering — because it is easy to defer and its absence is invisible until election day. Put it on the calendar with the specific scenarios, the pass/fail criteria, and the personnel required. Scheduled chaos engineering happens; intended chaos engineering does not.
"Election infrastructure is the one class of critical system where the failure consequences are simultaneously technical, operational, and constitutional. A voter registration database that is unavailable on Election Day does not just fail its SLO — it potentially disenfranchises voters whose constitutional right to participate depends on that system being available when they arrive at the polls. Site Reliability Engineering is the discipline that ensures the technical layer of democratic participation is treated with the engineering rigour that its constitutional significance demands."
Top comments (0)