ModSecurity and the OWASP Core Rule Set: How WAF Rules Actually Work

#security #webdev #opensource #devops

A Web Application Firewall inspects HTTP requests and responses against a set of rules, blocking requests that match known attack patterns before they reach your application. The mechanics of how this inspection works — and why false positives happen — are worth understanding before you deploy one in production.

ModSecurity: the engine

ModSecurity is the most widely deployed open-source WAF engine. It operates as a module inside Apache, Nginx, or IIS, or as a standalone reverse proxy component. It processes requests through a pipeline of phases:

Request headers phase: Rules run against request headers — User-Agent, Host, Content-Type, referer.
Request body phase: Rules run against the decoded request body — POST parameters, JSON payloads, file uploads.
Response headers phase: Rules run against response headers.
Response body phase: Rules run against the response body (enabled explicitly — high performance cost).
Logging phase: Final audit logging.

Rules are expressed in ModSecurity's SecRule syntax:

SecRule REQUEST_URI "@rx (?i:(\%27)|(')|(\-\-)|(\%23)|(#))" \
    "id:1001,phase:1,deny,status:403,\
     msg:'SQL Injection attempt detected',\
     logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}'"

This rule uses a regex match (@rx) against the REQUEST_URI variable. On match, it denies the request with HTTP 403 and logs the matched data.

The OWASP Core Rule Set

The OWASP Core Rule Set (CRS) is the standard ruleset for ModSecurity, covering the OWASP Top 10. Rather than a flat list of rules, it uses a scoring system:

Anomaly scoring mode: Each rule match adds to an anomaly score. When the cumulative score exceeds a threshold (default: 5 for inbound, 4 for outbound), the request is blocked. This reduces false positives from single-rule triggers while catching multi-signal attacks.

Rule A matches → score += 3
Rule B matches → score += 2
Total score 5 → BLOCK

Paranoia levels: CRS defines four paranoia levels (PL1–PL4). Higher levels activate more aggressive rules that catch more attacks but generate more false positives. PL1 is the default safe starting point; PL4 is suitable only for high-security applications with active rule tuning.

SQL injection detection (example)

CRS SQL injection rules (9xx series) detect patterns like:

' OR '1'='1
'; DROP TABLE users; --
UNION SELECT null, username, password FROM users

Detection uses multiple operators — regex matching, string operators, and libinjection (a dedicated SQL injection parser) for more reliable detection than pure regex.

XSS detection (example)

CRS XSS rules detect script injection patterns including:

<script>alert(1)</script>
<img src=x onerror=alert(1)>
javascript:void(0)

HTML entity encoding, URL encoding, and Unicode escapes are decoded before matching, preventing simple evasion via encoding.

False positives and tuning

False positives are the primary operational challenge with WAFs. Common sources:

Rich text editors: WYSIWYG editors submit HTML in POST bodies, triggering XSS rules.
SQL-like parameter values: Legitimate data containing apostrophes or SQL keywords (e.g., O'Brien) triggers SQL injection rules.
Large request bodies: File uploads or API payloads may trigger body size or content rules.

The correct tuning approach is targeted exclusions, not blanket rule disabling:

# Exclude rule 941100 (XSS) for the /editor/submit route only
SecRuleUpdateTargetById 941100 "!REQUEST_URI:/editor/submit"

# Exclude the 'content' parameter from SQL injection rules
SecRuleUpdateTargetById 942100 "!ARGS:content"

CacheGuard deploys ModSecurity with the OWASP CRS pre-configured at PL1, integrated after SSL termination in the reverse proxy pipeline so it inspects decrypted HTTPS traffic. Rule exclusions are manageable via the web interface without editing config files manually.

→ https://www.cacheguard.com/what-is-a-waf/

Originally published on the CacheGuard Blog. CacheGuard is free and open source — GitHub.