DEV Community: Aomi Qaza

Linux Auditd Blueprint: Real-Time Kernel Event Tracking & Security Auditing

Aomi Qaza — Fri, 14 Aug 2026 13:39:35 +0000

Linux Auditd Blueprint: Real-Time Kernel Event Tracking & Security Auditing

Security guide for configuring Linux Audit Framework (auditd) to log execve system calls, file integrity changes, and privilege escalation events.

Executive Summary & Key Takeaways

Execve Process Auditing: Track all executed commands across system users.
File Integrity Monitoring: Monitor write and attribute modifications on /etc/passwd and /etc/sudoers.
Audit Ruleset Modularization: Structure rules inside /etc/audit/rules.d/.
Log Analysis via ausearch: Query security event logs efficiently with aureport and ausearch.

1. Auditd Framework Architecture & Kernel Interception

The Linux Audit Subsystem is embedded directly inside the Linux kernel. It hooks system calls at the kernel boundary, capturing process executions, file access attempts, network socket creations, and user authentication events before processes complete.

Unlike standard syslog daemons (which depend on applications choosing to log messages), auditd intercepts events unconditionally at the system call level, making it tamper-resistant against user-space log manipulation.

The user-space auditd daemon collects events from kernel netfilter buffers and writes them to /var/log/audit/audit.log.

This granular logging capability forms the core foundation of Digital Forensics and Incident Response (DFIR) on Linux servers.

# Install auditd on Debian/Ubuntu
systemctl enable --now auditd

# Verify audit daemon status
auditctl -s

2. Auditing Process Execution (execve) across All Users

Attackers who gain initial shell access execute reconnaissance commands (such as whoami, id, uname -a, netstat). Standard bash history can be deleted or bypassed by executing binaries directly.

Adding execve audit rules records every single binary execution across the entire system, capturing exact command-line arguments, working directories, parent PIDs, and effective user IDs (EUID).

Filter by arch=b64 to capture 64-bit system calls efficiently.

Capturing full command-line arguments provides immutable forensic evidence during post-compromise investigations.

# Track 64-bit and 32-bit execve system calls
-a always,exit -F arch=b64 -S execve -k process_exec
-a always,exit -F arch=b32 -S execve -k process_exec

3. File Integrity Monitoring (FIM) for Critical System Files

File Integrity Monitoring (FIM) rules watch critical system configuration files for write (w), read (r), execute (x), or attribute change (a) operations.

Configure file watches on /etc/passwd, /etc/shadow, /etc/sudoers, and /etc/pam.d/ to alert DFIR incident responders immediately if account credentials or privilege policies are modified.

Assign custom key tags (-k ) to audit rules to facilitate rapid filtering in log analysis tools.

Monitoring permissions changes on SSH configuration files (/etc/ssh/sshd_config) prevents unauthorized backdoor insertion.

# File Integrity Watch Rules
-w /etc/passwd -p wa -k identity_changes
-w /etc/shadow -p wa -k identity_changes
-w /etc/sudoers -p wa -k privilege_changes
-w /etc/sudoers.d/ -p wa -k privilege_changes
-w /var/log/tallylog -p wa -k auth_logs

4. Structuring Modular Audit Rules in /etc/audit/rules.d/

Modern Linux distributions manage audit rules through modular files in /etc/audit/rules.d/*.rules, which augments auditctl when auditd starts.

Organize rules into numbered files (e.g., 10-base.rules, 30-fim.rules, 90-finalize.rules) to maintain clear rule precedence.

Enforce rule immutability by appending -e 2 as the final directive, preventing attackers from disabling audit rules without a reboot.

Immutable rules guarantee log integrity even if an attacker achieves temporary root privileges.

# /etc/audit/rules.d/99-finalize.rules
# Lock audit rules until next reboot
-e 2

5. Performance Tuning & Buffer Backlog Management

Under high system load, kernel audit buffers can overflow if the auditd daemon cannot write to disk fast enough, causing kernel event drops.

Tune kernel backlog buffer limits (-b) and set failure response actions (-f) in /etc/audit/rules.d/10-base.rules to handle burst traffic gracefully.

Configuring backlog buffers to 8192 prevents event drops during bursty web application workloads.

# /etc/audit/rules.d/10-base.rules
-D
-b 8192
-f 1

6. Incident Response & Log Querying via ausearch & aureport

The audit log file (/var/log/audit/audit.log) contains dense key-value pairs. Use built-in utilities ausearch and aureport for forensic queries.

Use aureport --summary to generate high-level executive reports on failed authentications, executable events, and file access violations.

Combine ausearch with jq or SIEM log forwarders (Elastic Filebeat, Vector) to stream structured audit events to central security operating centers.

# Query process execution events by key tag
ausearch -k process_exec -i

# Generate summary report of failed login attempts
aureport -l --failed

# Query modifications to /etc/sudoers
ausearch -k privilege_changes -i

Frequently Asked Questions (FAQ)

Q: Does auditd affect system CPU performance?

With optimized filter rules (-F arch=b64), auditd overhead is less than 1-2% CPU under heavy workloads.

Q: What does -e 2 mean in audit rules?

-e 2 locks the audit configuration permanently until system reboot, preventing an attacker who gains root from disabling audit logging.

Originally published at https://zyekh.com/blog/auditd-kernel-event-monitoring-and-dfir-logging.html

Rust in the Linux Kernel: Securing Core Subsystems & Memory Safety

Aomi Qaza — Fri, 14 Aug 2026 12:24:26 +0000

Rust in the Linux Kernel: Securing Core Subsystems & Memory Safety

Comprehensive engineering guide on how Rust eliminates spatial and temporal memory vulnerabilities in modern Linux kernel drivers and network modules.

Executive Summary & Key Takeaways

Memory Safety Enforcement: Eliminate use-after-free, double-free, and buffer overflow vulnerabilities at compile time.
Safe Abstraction Layers: Use C bindings wrapped in safe Rust traits to prevent undefined behavior in kernel modules.
Strict Concurrency Control: Utilize Rust ownership rules to eliminate data races in multi-threaded kernel worker threads.
Kernel Driver Modernization: Write Linux network and device drivers using idiomatic safe Rust abstractions.

1. The Memory Safety Paradigm in Kernel Development

Historically, over 70% of high-severity vulnerabilities in the Linux kernel stem from C memory safety flaws, including spatial out-of-bounds access, temporal use-after-free conditions, uninitialized memory reads, and race conditions in interrupt handlers. As kernel complexity expands to support high-throughput cloud infrastructure and heterogenous multi-core hardware, manual memory management in C becomes increasingly unsustainable for security engineering teams.

Rust introduces compile-time memory safety guarantees without relying on garbage collection algorithms or runtime execution overhead. By enforcing strict ownership rules, borrow checking, and explicit variable lifetimes, the Rust compiler proves the absolute absence of memory safety bugs before kernel code is ever assembled into binary ELF object files.

In the Linux kernel tree, Rust infrastructure complements existing C subsystems by allowing new device drivers, network protocol stacks, and virtual file system abstractions to be authored in safe Rust code, isolating raw pointer dereferences to audited, explicitly documented unsafe code blocks.

By mandating that all memory allocations and pointer operations pass strict static analysis, Rust eliminates entire classes of Common Vulnerabilities and Exposures (CVEs) that have historically plagued Linux kernel deployments across enterprise data centers.

Implementing automated continuous monitoring across production nodes ensures that compliance policies remain enforced during infrastructure updates.

Regular security audits should be integrated into DevOps CI/CD pipelines to verify that system configurations conform to zero-trust architecture standards.

Documenting system architecture and access control rules facilitates compliance verification during independent third-party security audits.

Enforcing strict runtime isolation boundaries prevents privilege escalation vectors across multi-tenant cloud environments.

// Example: Safe Linux Kernel Module in Rust
use kernel::prelude::*;

module! {
    type: RustKernelDemo,
    name: "rust_kernel_security",
    author: "Zyekh Abdul Qadir Jailani",
    description: "\"Rust Memory Safety Kernel Subsystem\","
    license: "GPL",
}

struct RustKernelDemo;

impl kernel::Module for RustKernelDemo {
    fn init(_name: &'static CStr, _module: &'static ThisModule) -> Result {
        pr_info!("Rust kernel module initialized with strict memory safety\n");
        Ok(RustKernelDemo)
    }
}

2. Data Race Elimination & Concurrency Safety

Data races in kernel space occur when two execution contexts (such as hardware interrupt service routines or concurrent SMP worker threads) access the exact same memory location simultaneously without synchronization, where at least one thread executes a write operation. Data races cause subtle kernel memory corruption and non-deterministic panics that are notoriously difficult to capture in production.

Rust guarantees data race freedom at compile time through the type system using the Send and Sync auto traits. A type is automatically marked as Send if ownership of its underlying data can be safely transferred across execution thread boundaries, while a type is Sync if references to it can be shared concurrently across multiple processing cores.

When developing Linux kernel device drivers in Rust, synchronization primitives such as Mutex and SpinLock wrap data types directly (e.g., Mutex). This structural encapsulation ensures that accessing inner state is physically impossible without acquiring the lock guard, which automatically releases the lock when going out of scope.

Furthermore, Rust's borrow checker enforces the aliasing XOR mutability rule: data may have either multiple immutable references (&T) or exactly one mutable reference (&mut T) at any given moment, permanently preventing race conditions during concurrent state updates.

Implementing automated continuous monitoring across production nodes ensures that compliance policies remain enforced during infrastructure updates.

Regular security audits should be integrated into DevOps CI/CD pipelines to verify that system configurations conform to zero-trust architecture standards.

Documenting system architecture and access control rules facilitates compliance verification during independent third-party security audits.

Enforcing strict runtime isolation boundaries prevents privilege escalation vectors across multi-tenant cloud environments.

// Safe Concurrency Control in Kernel Driver State
use kernel::sync::Mutex;

struct DriverState {
    packets_processed: u64,
    is_active: bool,
}

struct SafeDevice {
    state: Mutex,
}

impl SafeDevice {
    fn update_stats(&self) {
        let mut guard = self.state.lock();
        guard.packets_processed += 1;
    } // Mutex guard drops automatically, releasing lock
}

3. C Interoperability & Safe Wrapper Design

Integrating Rust into a multi-million-line C kernel codebase requires robust Foreign Function Interface (FFI) interoperability infrastructure. The kernel build system utilizes bindgen to generate raw Rust bindings directly from C header files, while safe Rust abstractions encapsulate raw C pointers inside safe types.

Designing safe kernel wrappers involves isolating raw pointer dereferences to audited, minimalist unsafe blocks. Once the public Rust wrapper API is proven mathematically sound, external callers can consume kernel functions without any risk of triggering undefined behavior or memory corruption.

This modular architecture enables an incremental modernization strategy: legacy C subsystems remain intact while newly developed hardware drivers, eBPF helper extensions, and security-critical modules are authored exclusively in safe Rust.

Modern Linux distributions are increasingly adopting LLVM toolchain builds, enabling Link-Time Optimization (LTO) between C and Rust compilation units to eliminate cross-language function call overhead entirely.

Implementing automated continuous monitoring across production nodes ensures that compliance policies remain enforced during infrastructure updates.

Regular security audits should be integrated into DevOps CI/CD pipelines to verify that system configurations conform to zero-trust architecture standards.

Documenting system architecture and access control rules facilitates compliance verification during independent third-party security audits.

Enforcing strict runtime isolation boundaries prevents privilege escalation vectors across multi-tenant cloud environments.

# Build kernel with Rust support enabled
make LLVM=1 rustavailable
make LLVM=1 menuconfig
# Enable CONFIG_RUST=y in Kernel Hacking -> Rust support
make LLVM=1 -j$(nproc)

4. Verification & Kernel Security Audit Checklist

Auditing Rust kernel modules requires verifying that every unsafe block contains an explicit // SAFETY: rationale comment explaining why the invariants cannot be violated by callers. Static analysis tools like Clippy and KASAN (Kernel Address Sanitizer) enforce strict coding standards.

Ensure that error handling in Rust kernel modules relies strictly on the kernel Result type instead of unwinding panics. Unwinding across FFI boundaries is undefined behavior in C, so Rust kernel code must use panic=abort configuration.

Automated CI/CD security pipelines should execute Clippy lints with warning-as-error flags enabled, verifying that no unapproved unsafe blocks are introduced into upstream kernel pull requests.

Regular security audits must review macro expansions and generated FFI bindings to ensure kernel memory layouts remain binary-compatible across architecture targets.

Implementing automated continuous monitoring across production nodes ensures that compliance policies remain enforced during infrastructure updates.

Regular security audits should be integrated into DevOps CI/CD pipelines to verify that system configurations conform to zero-trust architecture standards.

Documenting system architecture and access control rules facilitates compliance verification during independent third-party security audits.

Enforcing strict runtime isolation boundaries prevents privilege escalation vectors across multi-tenant cloud environments.

# Verify Rust kernel code with Clippy and KASAN
make LLVM=1 CLIPPY=1 path/to/module.o

# Audit kernel logs for Rust initialization
dmesg | grep -i "Rust"

Frequently Asked Questions (FAQ)

Q: Does Rust add runtime overhead to the Linux kernel?

No. Rust compiles directly to native machine code via LLVM without garbage collection or runtime overhead, yielding performance identical to C.

Q: Can Rust completely replace C in the Linux kernel?

No. Rust is designed to coexist alongside C. Core kernel architecture remains in C while drivers, filesystems, and security modules leverage Rust.

Originally published at https://zyekh.com/blog/rust-in-linux-kernel-security-and-memory-safety-blueprint-2026.html

vLLM PagedAttention: Memory Optimization & High-Throughput LLM Inference Tuning

Aomi Qaza — Fri, 14 Aug 2026 12:18:41 +0000

vLLM PagedAttention: Memory Optimization & High-Throughput LLM Inference Tuning

Deep dive into OS-inspired virtual memory management for LLM KV caches, maximizing GPU utilization, and achieving 20x throughput scaling for production inference.

Executive Summary & Key Takeaways

KV Cache Bottleneck: Understand why traditional LLM inference wastes up to 80% of GPU memory due to fragmentation.
PagedAttention Mechanism: Inspired by OS virtual memory, allocate KV cache non-contiguously to eliminate memory waste.
High-Throughput Tuning: Optimize vLLM batch sizes, tensor parallelism, and quantization for massive concurrent requests.

1. The Anatomy of the KV Cache Bottleneck

In auto-regressive Transformer models, generating the next token requires attending to all previously generated tokens. Recomputing these attention scores for every new token is computationally prohibitive. Therefore, inference engines cache the Key (K) and Value (V) tensors for past tokens. This is known as the KV cache.

As sequence lengths grow, the KV cache expands linearly, consuming massive amounts of high-bandwidth memory (HBM) on the GPU. In traditional inference frameworks, memory for the KV cache is allocated statically and contiguously based on the maximum possible sequence length of the request.

Because the actual generation length is unpredictable, this static allocation leads to severe internal fragmentation. A request might reserve memory for 2048 tokens but only generate 20 tokens. Additionally, external fragmentation occurs as requests of varying lengths interleave, creating unusable gaps in memory.

Profiling reveals that in naive deployments, up to 80% of GPU memory dedicated to the KV cache is wasted. This memory starvation prevents the engine from batching more concurrent requests, severely limiting the overall throughput of the inference server, regardless of how much raw compute power the GPU possesses.

Addressing this bottleneck requires a fundamental shift in how GPU memory is managed during the decoding phase of LLM inference.

# Traditional static allocation (Pseudocode)
kv_cache = allocate_gpu_memory(batch_size, num_heads, max_seq_len, head_size)
# Wastes memory if actual_seq_len << max_seq_len

2. PagedAttention: OS Virtual Memory for LLMs

vLLM introduces PagedAttention, an algorithm that elegantly solves the KV cache fragmentation problem by borrowing concepts from operating system virtual memory management. Instead of allocating memory contiguously, PagedAttention divides the KV cache into fixed-size blocks (pages).

Each block contains the KV vectors for a fixed number of tokens. When a request is processed, the engine dynamically allocates these blocks on demand. The logical blocks associated with a specific request are mapped to non-contiguous physical blocks in GPU memory via a block table, mirroring how an OS maps virtual pages to physical frames.

During the attention computation, the PagedAttention kernel fetches the KV vectors by traversing the block table. Because the blocks do not need to be contiguous, external fragmentation is entirely eliminated. Internal fragmentation is restricted only to the final, partially filled block of a request.

This dynamic allocation allows the inference engine to pack significantly more concurrent requests into the same GPU memory footprint. Furthermore, it enables memory sharing across different requests. For example, if multiple requests share the same system prompt, the blocks containing the prompt's KV cache can be shared, drastically reducing memory consumption.

By near-optimally utilizing GPU memory, PagedAttention allows vLLM to achieve state-of-the-art throughput, outperforming traditional engines like Hugging Face Transformers by up to 24x in high-concurrency scenarios.

3. vLLM Deployment & Throughput Tuning

Deploying vLLM in a production environment requires careful tuning of its core parameters to maximize hardware utilization. The most critical configuration is the --gpu-memory-utilization flag. This dictates what percentage of the GPU's HBM is reserved for the KV cache pool versus the model weights.

For large models (e.g., Llama-3 70B) spanning multiple GPUs, Tensor Parallelism (TP) is essential. vLLM utilizes Megatron-LM's tensor parallel algorithms to shard the model's weight matrices across multiple devices. Configuring --tensor-parallel-size correctly ensures that the compute load is balanced and the inter-GPU communication overhead is minimized.

To further increase throughput, operators must tune the --max-num-batched-tokens and --max-num-seqs parameters. These dictate the aggressiveness of the continuous batching scheduler. Pushing these values too high can lead to GPU Out-Of-Memory (OOM) errors during the prefill phase, while setting them too low leaves compute resources idle.

Quantization is another powerful lever. vLLM supports AWQ (Activation-aware Weight Quantization) and GPTQ, allowing 16-bit models to be compressed into 4-bit representations. This drastically reduces the memory footprint of the model weights, freeing up more HBM for the PagedAttention KV cache pool, which directly translates to higher concurrency.

Finally, enabling CUDA Graph capture for the decoding phase eliminates CPU dispatch overhead, significantly reducing latency for small batch sizes. Tuning these parameters in tandem transforms a standard GPU node into a high-octane inference engine.

# Starting vLLM server with optimized parameters for production
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.90 \
    --max-num-batched-tokens 8192 \
    --quantization awq

4. Continuous Batching and Iteration-Level Scheduling

Traditional inference engines use static batching, where a batch of requests is processed together, and the engine must wait for the longest request in the batch to complete before accepting new requests. This leads to massive idle times for early-finishing requests.

vLLM employs continuous batching (or iteration-level scheduling). The scheduler evaluates the state of all requests at every single token generation step. As soon as a request completes, its KV cache blocks are instantly freed, and a new request is immediately injected into the active batch.

This fine-grained scheduling, coupled with PagedAttention's dynamic memory management, ensures that the GPU remains fully saturated at all times. The continuous influx and eviction of requests create a steady-state pipeline that maximizes overall system throughput.

When deployed behind an OmniRouter gateway, vLLM nodes provide a highly predictable, high-throughput backend capable of absorbing massive traffic spikes without catastrophic latency degradation.

The combination of OmniRouter's intelligent traffic shaping and vLLM's ruthless hardware optimization represents the pinnacle of modern AI engineering.

Frequently Asked Questions (FAQ)

Q: Can vLLM run on consumer-grade GPUs?

Yes, vLLM supports consumer GPUs (e.g., RTX 3090/4090) provided the model weights and the configured KV cache pool fit within the available VRAM (e.g., 24GB). Quantization is highly recommended for consumer hardware.

Originally published at https://zyekh.com/blog/vllm-pagedattention-high-throughput-inference-tuning.html

High-Speed eBPF/XDP Packet Filtering for Linux Server DDoS Mitigation

Aomi Qaza — Fri, 14 Aug 2026 12:18:11 +0000

High-Speed eBPF/XDP Packet Filtering for Linux Server DDoS Mitigation

Technical guide for writing eBPF eXpress Data Path (XDP) kernel programs to drop malicious DDoS traffic at the network driver level.

Executive Summary & Key Takeaways

XDP_DROP Early Decision: Drop malicious UDP/SYN floods before allocating sk_buff memory.
Kernel Map Invalidation: Dynamic IP blocklists via eBPF BPF_MAP_TYPE_HASH maps.
Zero-Copy Performance: Process 10M+ packets per second on commodity server hardware.
Clang/LLVM BPF Compilation: Build C programs directly into BPF bytecode targets.

1. Understanding XDP Architecture vs Traditional Linux SKB Allocation

Standard Linux network processing allocates a complex kernel socket buffer data structure (sk_buff) for every incoming packet before firewall rules (iptables/nftables) can evaluate the packet.

Under volumetric DDoS attacks (such as 10 Million Packets Per Second UDP floods), the CPU time spent allocating and freeing sk_buff structures exhausts kernel memory and CPU cache lines, causing severe packet drops and server unresponsiveness.

eXpress Data Path (XDP) provides a high-performance bare-metal packet processing framework. XDP programs execute eBPF bytecode directly inside the network driver's RX ring buffer before sk_buff memory allocation occurs.

# Inspect network interface driver XDP support
ip link show eth0

2. XDP Packet Processing Actions (XDP_DROP vs XDP_PASS)

An XDP program evaluates raw packet data directly from driver memory and returns one of five verdict codes to the network card driver:

XDP_DROP: Immediately recycles the packet buffer in the driver RX ring without allocating memory or notifying the CPU TCP/IP stack.

XDP_PASS: Passes the packet up to the normal Linux TCP/IP network stack for standard processing.

XDP_TX: Bounces the packet back out the same network interface it arrived on (useful for high-speed load balancers).

// XDP Action Constants
// XDP_DROP = 1
// XDP_PASS = 2
// XDP_TX   = 3

3. Writing a Production XDP Packet Filter in C

XDP C code uses eBPF helpers and header pointers to parse Ethernet, IPv4, and UDP/TCP protocol headers safely.

The eBPF verifier verifies memory bounds checking before loading bytecode into the kernel, ensuring the XDP program can never crash the Linux kernel.

The following C program parses incoming IPv4 headers and drops packets matching blacklisted source IP addresses stored in an eBPF hash map:

#include 
#include 
#include 
#include 

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 100000);
    __type(key, __be32);
    __type(value, __u64);
} blacklist SEC("maps");

SEC("xdp")
int xdp_firewall(struct xdp_md *ctx) {
    void *data_end = (void *)(long)ctx->data_end;
    void *data = (void *)(long)ctx->data;
    struct ethhdr *eth = data;

    if ((void *)(eth + 1) > data_end) return XDP_PASS;
    if (eth->h_proto != __constant_htons(ETH_P_IP)) return XDP_PASS;

    struct iphdr *iph = (void *)(eth + 1);
    if ((void *)(iph + 1) > data_end) return XDP_PASS;

    __be32 src_ip = iph->saddr;
    __u64 *value = bpf_map_lookup_elem(&blacklist, &src_ip);
    if (value) {
        return XDP_DROP;
    }
    return XDP_PASS;
}
char _license[] SEC("license") = "GPL";

4. Compiling & Loading Bytecode Targets via Clang/LLVM

Compile XDP C code into BPF Executable and Linkable Format (ELF) targets using Clang and LLVM compiler toolchains.

Attach the compiled BPF bytecode to a network interface using standard iproute2 ip link commands.

# Compile C code to BPF bytecode
clang -O2 -target bpf -c xdp_firewall.c -o xdp_firewall.o

# Attach XDP program to eth0 network interface
ip link set dev eth0 xdp obj xdp_firewall.o sec xdp

# Detach XDP program from eth0
ip link set dev eth0 xdp off

5. Dynamic Blocklist Management via BPF Maps

BPF Maps are high-speed shared memory structures bridging kernel space and user-space daemons.

User-space monitoring daemons (such as Fail2ban or custom Go/Python agents) populate blocked IP addresses into the BPF map dynamically without reloading the XDP program.

# Add blocked IP (e.g. 198.51.100.45) to BPF map via bpftool
bpftool map update id 4 key 198 51 100 45 value 1 0 0 0 0 0 0 0

# Dump current BPF map contents
bpftool map dump id 4

6. High-Throughput Packet Benchmark Verification

Verify XDP packet drop counters using ethtool or bpftool stats.

Benchmark packet processing throughput under simulated UDP floods using pktgen.

# Inspect XDP active interface stats
ip -s link show dev eth0

# View eBPF program list
bpftool prog show

Frequently Asked Questions (FAQ)

Q: What is the difference between XDP_DROP and iptables DROP?

XDP_DROP drops packets in driver memory before Linux creates socket buffers, yielding 10x higher throughput.

Q: Does XDP require special network card hardware?

No. XDP supports native mode (driver level), offloaded mode (SmartNIC hardware), and generic mode (fallback for any network driver).

Originally published at https://zyekh.com/blog/ebpf-xdp-packet-filtering-and-ddos-mitigation.html

Zero-Trust SSH Access Blueprint: FIDO2 Hardware Keys & SSH Certificate Authority

Aomi Qaza — Fri, 14 Aug 2026 03:21:24 +0000

Zero-Trust SSH Access Blueprint: FIDO2 Hardware Keys & SSH Certificate Authority

Comprehensive technical blueprint for securing Linux SSH infrastructure using FIDO2/YubiKey hardware tokens, SSH Certificate Authority (SSH CA), and short-lived certificates.

Executive Summary & Key Takeaways

Eliminate Static Keys: Migrate from static authorized_keys deployment to short-lived SSH Certificates.
FIDO2 Hardware Bound: Enforce ed25519-sk key pairs tied to physical security tokens (YubiKey/FIDO2).
Centralized Authority: Use an offline SSH Certificate Authority (CA) to sign user access requests with automatic 8-hour expiration.
Zero Administrative Sprawl: Adding or revoking user permissions requires zero modifications on target servers.

Traditional SSH key management across growing server fleets suffers from a critical flaw: static public key sprawl. Managing thousands of ~/.ssh/authorized_keys files across production instances creates massive administrative overhead, increases the blast radius of compromised developer workstations, and makes offboarding security audits nearly impossible.

A true Zero-Trust SSH Access Model replaces static SSH keys with two cryptographic pillars:

FIDO2 / Security Key Hardware Tokens (ed25519-sk): Private key material never leaves the physical YubiKey token and requires physical touch plus user PIN.
SSH Certificate Authority (SSH CA): Short-lived SSH certificates (e.g., valid for 8 hours) signed by a centralized CA key, eliminating manual authorized_keys deployment.

1. The Problem with Static SSH Public Keys

In standard SSH deployments, when an engineer needs access to a production server, their public key is appended to the server's authorized_keys file. Over time, this leads to significant vulnerabilities:

No Expiration: Public keys remain valid indefinitely until manually purged.
Workstation Compromise: Unprotected SSH private keys stored on developer disk drives can be exfiltrated by malware.
Lack of Centralized Revocation: Revoking access requires executing cleanup scripts across every instance.

2. Hardware Security Keys: OpenSSH FIDO2 / U2F

Since OpenSSH 8.2, native support for security keys (FIDO2 / U2F) is supported via the ed25519-sk and ecdsa-sk key types. Key generation requires the physical hardware token connected to the machine.

# Generate an SSH key backed by a hardware token with user-presence verification
ssh-keygen -t ed25519-sk -O touch-required -C "engineer@company.com (YubiKey)"

[ NOTE ] For enhanced security, generate resident keys (discoverable credentials) with PIN protection so the key stub can be retrieved directly from the token on new devices:

# Generate a resident key with mandatory PIN and touch requirement
ssh-keygen -t ed25519-sk -O resident -O verify-required -C "engineer@company.com (Resident)"

3. Setting Up a Centralized SSH Certificate Authority

Instead of copying individual public keys to every target machine, servers are configured to trust a single CA Public Key. The CA signs engineer public keys with short expiration times and restricted permissions.

Step A: Generate the CA Key Pair

Generate a secure host CA key on an isolated offline machine or Hardware Security Module (HSM):

# Generate the SSH Certificate Authority key pair
ssh-keygen -t ed25519 -f /etc/ssh/ca/ssh_user_ca -C "Production SSH User CA 2026"

Step B: Configure Target Linux Servers to Trust the CA

On all Linux production servers, place the CA public key in /etc/ssh/ssh_user_ca.pub and update /etc/ssh/sshd_config:

# Append to /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/ssh_user_ca.pub
AuthorizedPrincipalsFile /etc/ssh/authorized_principals/%u

Create principal definitions for target system accounts (e.g., /etc/ssh/authorized_principals/ubuntu containing the line admin-role).

Step C: Signing User Keys with Expiration & Principals

When an engineer requests access, the CA signs their public key to produce a certificate (id_ed25519_sk-cert.pub):

# Sign engineer's public key valid for 8 hours for principal 'admin-role'
ssh-keygen -s /etc/ssh/ca/ssh_user_ca \
  -I "engineer@company.com" \
  -n "admin-role" \
  -V +8h \
  -z 20260803001 \
  id_ed25519_sk.pub

4. Related Privacy & Security Tools

To assist with infrastructure configuration, security audits, and key validation, use these privacy-first client-side web tools:

Secure Password & Passphrase Generator -> Generate high-entropy PINs for hardware tokens.
Cryptographic Hash Generator -> Verify SHA-256 fingerprints of SSH public keys and certificates.
Side-by-Side Diff Checker -> Audit differences between server sshd_config templates.
Environment Variables & Secrets Formatter -> Format SSH environment variables safely in-browser.

5. Verification & Security Audit Checklist

Verify that your setup enforces Zero-Trust principles using the following checklist:

# Inspect certificate properties and expiration details
ssh-keygen -L -f id_ed25519_sk-cert.pub

# Test SSH login using explicit certificate credential
ssh -i id_ed25519_sk -i id_ed25519_sk-cert.pub engineer@server.company.com

Frequently Asked Questions (FAQ)

Q: What happens when an SSH certificate expires?

Once the certificate validity period expires (e.g., +8h), OpenSSH automatically rejects all authentication attempts using that certificate. No server-side cleanup or manual key removal is required.

Q: Can FIDO2 SSH keys be used without internet connectivity?

Yes. OpenSSH communicates directly with the FIDO2 hardware token over USB/NFC via standard libfido2 drivers, requiring zero external internet connection.

Originally published at https://zyekh.com/blog/zero-trust-ssh-access-with-fido2-and-ssh-ca.html