DEV Community

Cover image for Enterprise Linux Kernel Tuning, eBPF & Hardening Guide
Amir Mehmood
Amir Mehmood

Posted on Originally published at hostingpodium.com

Enterprise Linux Kernel Tuning, eBPF & Hardening Guide

#kubernetes #devops #cloud #linux

Originally published on the HostingPodium Enterprise Engineering Blog.

Free SRE & Developer Diagnostics: Test your stack live with our DMARC Record Checker, Generator & SPF/DKIM Validator, Free Website Speed & Core Web Vitals Checker, or explore all 9 Free Webmaster & Cloud Engineering Tools.

This guide delivers a step-by-step reference architecture for Enterprise Linux in cloud environments, covering kernel tuning, storage layout, security hardening, and operational runbooks. It equips engineers with concrete configurations and clear trade-off analyses for production deployments.

Introduction

This guide presents a production-grade reference architecture for Enterprise Linux workloads running on public or private cloud infrastructure. It assumes a baseline of RHEL 9 (kernel 5.14) with extended support contracts, but the principles apply to any binary-compatible distribution.

Scope

The document covers kernel tuning, compute provisioning, storage layout, network stack, configuration management, observability, security hardening, high-availability design, disaster-recovery procedures, and a concrete runbook for incident response.

Baseline Platform Selection

Three common Enterprise Linux variants dominate cloud deployments:

  • Red Hat Enterprise Linux (RHEL) 9 – 10-year lifecycle, certified images for AWS, Azure, GCP.

  • Oracle Linux 9 – compatible kernel, optional Unbreakable Enterprise Kernel (UEK) with 4.14-based performance patches.

  • CentOS Stream 9 – rolling preview of RHEL, suitable for development but not recommended for SLA-bound production.

When selecting a variant, compare the following metrics:

Metric RHEL 9 Oracle Linux 9 CentOS Stream 9
Support window 10 years 10 years (optional UEK) Rolling
Kernel version 5.14 5.14 / UEK 4.14 5.14-rc
Certification Full cloud provider certification Partial certification None

RHEL 9 remains the safest choice for workloads with strict SLA requirements.

Kernel Parameter Baselines

Baseline /etc/sysctl.d/99-enterprise-linux.conf should contain:

# Networking
net.core.somaxconn = 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.ip_local_port_range = 1024 65535
# Virtual memory
vm.swappiness = 10
vm.dirty_ratio = 15
vm.dirty_background_ratio = 5
# File system
fs.file-max = 2097152
fs.nr_open = 1048576
Enter fullscreen mode Exit fullscreen mode

Apply with sysctl --system. These values have been validated on m5.large (2 vCPU, 8 GiB) instances handling 10 k concurrent connections.

Compute Provisioning

Cloud VM sizing must respect NUMA boundaries to avoid cross-node memory latency. Use the following pattern for instance families that expose multiple sockets:

# Example for a 4-socket, 64 vCPU instance
cloud-init:
 cpu_affinity:
 - socket: 0
 cores: 0-15
 - socket: 1
 cores: 16-31
 - socket: 2
 cores: 32-47
 - socket: 3
 cores: 48-63
Enter fullscreen mode Exit fullscreen mode

When the cloud provider does not expose socket topology, pin processes manually via taskset or cset and verify with numactl --hardware.

Storage Architecture

Enterprise workloads typically require a mix of block and file storage. The recommended layout:

  • Root volume: XFS, mkfs.xfs -f -d su=64k,sw=1, 30 GiB, encrypted with LUKS.

  • Data volume: LVM on top of NVMe SSDs, striped across all devices (e.g, lvcreate -i 4 -I 64K -L 500G -n data vg0).

  • Backup volume: EBS gp3 with provisioned IOPS 10 k, mounted via noatime,nodiratime.

Performance target: ≥ 2 GB/s sequential read on striped LVM, ≤ 0.5 ms latency for 4 KB random reads.

Network Stack

For latency-sensitive services, enable SR-IOV and offload features where supported. Example /etc/sysconfig/network-scripts/ifcfg-eth0 for an AWS ENA interface:

DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
HWADDR=02:7b:64:2e:1c:5a
ETHTOOL_OPTS="-K tx off rx off tso off gso off gro off"
Enter fullscreen mode Exit fullscreen mode

Bond two NICs in active-backup mode for redundancy:

DEVICE=bond0
TYPE=Bond
BONDING_OPTS="mode=1 miimon=100"
Enter fullscreen mode Exit fullscreen mode

Switch from iptables to nftables for reduced rule processing overhead. A minimal /etc/nftables.conf:

table inet filter {
 chain input {
 type filter hook input priority 0;
 policy drop;
 ct state established,related accept;
 iif "lo" accept;
 tcp dport { 22, 80, 443 } accept;
 }
}
Enter fullscreen mode Exit fullscreen mode

Configuration Management

Ansible remains the most widely adopted tool for declarative provisioning on Enterprise Linux. A sample playbook to enforce the kernel parameters shown earlier:

- hosts: all
 become: true
 tasks:
 - name: Deploy sysctl config
 copy:
 dest: /etc/sysctl.d/99-enterprise-linux.conf
 content: |
 net.core.somaxconn = 65535
 net.ipv4.tcp_tw_reuse = 1
 vm.swappiness = 10
 notify: Reload sysctl

 handlers:
 - name: Reload sysctl
 command: sysctl --system
Enter fullscreen mode Exit fullscreen mode

All roles must be version-locked in requirements.yml to guarantee repeatable builds across environments.

Observability Stack

Metrics collection relies on node_exporter (v1.6) and systemd-journal-gatewayd for log access. Prometheus scrape configuration for a fleet of 200 instances:

scrape_configs:
 - job_name: 'linux_nodes'
 static_configs:
 - targets: ['10.0.0.1:9100','10.0.0.2:9100',..]
 relabel_configs:
 - source_labels: [__address__]
 regex: '(.*):9100'
 target_label: instance
 replacement: '${1}'
Enter fullscreen mode Exit fullscreen mode

Alerting rule for CPU saturation:

- alert: CPUHighUtilization
 expr: avg by (instance) (rate(node_cpu_seconds_total{mode="user"}[5m])) > 0.85
 for: 2m
 labels:
 severity: critical
 annotations:
 summary: "CPU usage > 85% on {{ $labels.instance }}"
 runbook: "{{ .Values.runbookUrl }}/cpu_saturation.md"
Enter fullscreen mode Exit fullscreen mode

Log rotation is handled by logrotate with a 7-day retention policy for /var/log/messages and a 30-day policy for application logs.

Security Hardening

Enable SELinux in enforcing mode and lock down the policy to targeted with custom modules for container runtimes. Example /etc/selinux/config:

SELINUX=enforcing
SELINUXTYPE=targeted
Enter fullscreen mode Exit fullscreen mode

Auditd rule to capture all privileged command execution:

-w /usr/bin/sudo -p x -k privileged_exec
Enter fullscreen mode Exit fullscreen mode

Run OpenSCAP against the xccdf_org.ssgproject.content_profile_cis profile weekly. Non-compliant findings must be triaged within 48 hours.

High Availability Design

Two patterns dominate HA for Linux services:

  • Active-passive – keepalived VRRP provides a floating IP; Pacemaker manages resource failover.

  • Active-active – distributed databases (e.g, Galera) synchronize state; load balancer (HAProxy) performs health checks.

Sample keepalived configuration for a virtual IP 10.0.0.100:

vrrp_instance VI_1 {
 state MASTER
 interface eth0
 virtual_router_id 51
 priority 150
 advert_int 1
 authentication {
 auth_type PASS
 auth_pass secret123
 }
 virtual_ipaddress {
 10.0.0.100/24 dev eth0
 }
}
Enter fullscreen mode Exit fullscreen mode

Fencing is mandatory for split-brain protection. Use stonith:external/ipmi on bare-metal and cloud-provider-fence for virtualized nodes.

Disaster Recovery

Backup strategy combines snapshot-based recovery for block devices and file-level archiving for configuration files.

  • Daily EBS snapshots retained 30 days (RPO ≈ 12 h).

  • Nightly borg backup of /etc, /var/lib/mysql, and /opt/app to an S3 bucket (RPO ≈ 24 h, RTO ≈ 15 min).

Recovery test cadence: quarterly full-restore drills on a separate VPC.

Operational Runbook

Incident: CPU usage spikes above 90 % for more than 5 minutes.

  • Confirm alert in Prometheus UI.

  • SSH to the affected host (use bastion, MFA).

  • Run top -b -n1 | head -15 to identify offending process.

  • If the process is a known service, restart via systemctl restart service. Record the action in the ticket.

  • If the process is unknown or malicious, capture a core dump with coredumpctl dump -o /tmp/core.

  • Check for runaway threads using ps -eo pid,ppid,cmd,%cpu --sort=-%cpu | head.

  • Apply a temporary cpulimit rule if immediate mitigation is required.

  • After stabilization, review /var/log/audit/audit.log for privilege escalation attempts.

  • Update the runbook if a new pattern emerges.

Trade-off Summary

Design Choice Pros Cons Typical Use-Case
Active-passive keepalived Simple, low overhead Failover latency ~5 s Stateless web front-ends
Active-active Galera Zero-downtime writes Higher network traffic, complex conflict resolution Transactional databases
NVMe striped LVM Maximum sequential throughput Increased rebuild time if a device fails Data-lake ingest nodes
SR-IOV NIC Near-bare-metal latency Requires hypervisor support, reduces live-migration flexibility High-frequency trading front-ends

Choosing the right combination depends on SLA targets, cost constraints, and operational skill set.

For organizations looking to deploy bare-metal Linux infrastructure without virtualized CPU steal or noisy neighbor contention, explore our DigitalOcean Dedicated Alternative guide.

Free Webmaster, SRE & Cloud Diagnostic Tools (No Signup Required)

Verify and benchmark your production infrastructure directly from your browser using the HostingPodium Free Webmaster & Developer Suite:

Free Engineering Utility Primary Diagnostic Use Case
Global DNS Propagation & SSL Checker Verify live A, AAAA, CNAME, MX, TXT, NS & CAA records across 21+ global resolvers + TLS certificate audits
Website Speed Test & Core Web Vitals Auditor Measure server TTFB, LCP, INP, CLS, and HTTP/2 compression latency in real time
Rich Snippets & Schema.org JSON-LD Generator Generate & validate structured JSON-LD markup for TechArticles, FAQs, Products & Organizations
HTTP Redirect Chain & Status Code Tracer Trace 301/302 redirect hops, inspect response headers, and verify load balancer health checks
DMARC, SPF & DKIM Email Security Validator Audit domain email authentication policies and generate compliant DMARC records
Google SERP Simulator & Meta Tag Previewer Preview desktop/mobile Google snippets, OpenGraph cards, and pixel-width truncation
Free WebP Image Converter & Optimizer Client-side PNG/JPG to WebP compression for faster Core Web Vitals LCP

Related Enterprise Architecture & Benchmarks

Top comments (0)