DEV Community: Zepher Ashe

Stop Writing Python Like JavaScript: Common Language-Switching Mistakes

Zepher Ashe — Tue, 14 Apr 2026 22:18:06 +0000

1. Respect Language Idioms
2. Mind the Type System
3. Handle Errors the Right Way
4. Manage Dependencies Correctly
5. Be Conscious of Execution Context
6. Security Practices Don’t Always Translate
7. Testing & Tooling

Overview

When moving between languages, the main risks are forcing habits from one language onto another, misusing idioms, or forgetting environment-specific behaviours. The following practices help minimise friction and avoid subtle bugs.

1. Respect Language Idioms

Don’t force a main() everywhere.
- Bash, Perl, PHP, JavaScript → natural to write procedural top-down code.
- Python, Java, Go, Rust, C/C++ → explicit entry point is expected.
Follow the ecosystem’s coding style:
- Python → snake_case for variables/functions, CamelCase for classes.
- JavaScript → camelCase for variables/functions, PascalCase for classes.
- PowerShell → Verb-Noun (Get-Process).
- Java → CamelCase everywhere, verbose OOP structure.

2. Mind the Type System

Dynamic languages (Python, Bash, Perl, PHP, JavaScript):
- Expect runtime type errors; validate inputs early.
- Use linters/type checkers if available (mypy for Python, tsc for TypeScript).
Static languages (C, C++, Java, Go, Rust):
- Embrace compiler feedback; it saves runtime pain.
- Prefer explicit over inferred types when readability matters.
Hybrid (PowerShell):
- Strongly typed but still allows implicit conversions → always declare expected types in Param() blocks.

3. Handle Errors the Right Way

Bash/Perl: check $? or $! exit codes; use set -euo pipefail in Bash.
Python/Java/JavaScript/PHP: structured try/except or try/catch.
Rust: pattern-match on Result/Option.
Go: check returned err explicitly (if err != nil { ... }).
PowerShell: try/catch/finally with terminating errors.

4. Manage Dependencies Correctly

Python → pip/poetry/venv.
JavaScript → npm/yarn.
PHP → composer.
Go → go mod.
Rust → cargo.
Perl → CPAN/cpanm.
Avoid hardcoding library paths; use ecosystem tools for portability.

5. Be Conscious of Execution Context

CLI scripts: Bash, Perl, Python — make sure you handle arguments ($1, sys.argv, @ARGV).
Web scripts: PHP, JavaScript — expect interaction with HTTP state, superglobals, or events.
Compiled binaries: Go, Rust, C/C++ — distribute with care; consider static builds for portability.
Mixed models: PowerShell, Python, Node.js → scripts can be interactive or part of automation pipelines.

6. Security Practices Don’t Always Translate

String handling:
- Perl, Bash, PHP → watch out for injection risks (shell eval, SQL injection).
- Python, Go, Rust → encourage safer libraries by default.
Secrets handling:
- Never hardcode credentials; environment variables or vaults are standard.
Input validation:
- JavaScript/PHP → sanitize all external input (XSS/SQLi risks).
- Bash/Perl → validate before using in system commands.

7. Testing & Tooling

Always use linters/formatters when switching (helps enforce idioms):
- Python → black, flake8.
- JavaScript → eslint, prettier.
- Go → gofmt.
- Rust → clippy, rustfmt.
- PowerShell → PSScriptAnalyzer.
Write small unit tests in the ecosystem’s framework:
- Python → pytest.
- Java → JUnit.
- PHP → PHPUnit.
- Go → built-in go test.
- Rust → built-in cargo test.

Switching isn’t just syntax — it’s mindset. Learn the idioms, adopt ecosystem tools, and adjust error-handling and security practices to the language’s expectations.

Linux Observability Tools — A Practical Guide

Zepher Ashe — Mon, 09 Mar 2026 09:45:12 +0000

Linux Observability Tools

Observability is the ability to understand what a Linux system is doing internally by examining the signals it emits — metrics, logs, traces, and events.

This guide provides a structured overview of Linux observability tools, grouped by the system layers they inspect. It is designed as a practical reference for troubleshooting, performance engineering, capacity planning, and DevSecOps workflows.

🧱 1. Application & User-Space Observability
🧩 2. System Libraries & Syscall Interface
🧬 3. Kernel Subsystems Observability
🔩 4. Device Drivers & Block Layer Observability
📦 5. Storage & Swap Observability
🌐 6. Network Stack & NIC Observability
🖥️ 7. Hardware Observability (CPU, RAM, Buses)
📊 8. System-Wide Observability Tools

📊 Overview Diagram

Diagram © Brendan Gregg — used here with attribution for educational and informational purposes.

The diagram maps common observability tools to layers of the Linux operating system, from user-space applications down to hardware, providing a mental model for selecting the right tool during analysis or incident response.

🧱 1. Application & User-Space Observability

These tools inspect behaviour at the process and application level, including interactions with system libraries.

🔧 Tools

strace – traces system calls made by an application.
ltrace – traces dynamic library calls.
ss – modern socket statistics (replacement for netstat).
netstat – legacy but still useful for connection state overview.
sysdig – system-wide syscall/event capture and filtering.
lsof – lists open files, sockets, pipes, etc.
pidstat – per-process CPU, memory, I/O, threads.
pcstat – page cache statistics for specific files.

🧠 When to use

Debugging why an application is slow or blocked.
Identifying network usage per process.
Auditing open files and ports.
Understanding syscall patterns for performance tuning.

🧩 2. System Libraries & Syscall Interface

This layer sits between applications and the kernel. Tools here help examine transitions between user-space and kernel-space.

🔧 Tools

strace / ltrace – observe execution flow into syscalls and libraries.
perf – syscall latency, profiling, hotspots.
ftrace – built-in kernel tracer for syscalls and function calls.
SystemTap (stap) – programmable probes for syscalls.
LTTng – high-performance tracing for production systems.
eBPF / bpftrace – modern, safe kernel-level instrumentation.

🧠 When to use

Diagnosing syscall bottlenecks.
Monitoring unexpected kernel interactions.
High-resolution production tracing with low overhead (eBPF).

🧬 3. Kernel Subsystems Observability

The kernel handles filesystems, memory management, scheduling, and networking. Tools here inspect these internal mechanisms.

🔧 Tools

perf – scheduler behaviour, CPU cycles, kernel hotspots.
tcpdump – raw packet capture at the IP/Ethernet layers.
iptraf – lightweight network utilisation monitor.
vmstat – processes, memory, swap, I/O, interrupts.
slabtop – kernel slab allocator usage.
free – memory allocation breakdown.
pidstat – scheduler awareness and per-thread stats.
tiptop – per-thread metrics using hardware counters.

🧠 When to use

Identifying memory pressure, leaks, or slab exhaustion.
Determining network packet loss or congestion.
Analysing scheduler-induced latency.
Understanding kernel-side performance issues.

🔩 4. Device Drivers & Block Layer Observability

These tools examine I/O as it flows through the Linux block subsystem.

🔧 Tools

iostat – block device throughput and latency.
iotop – per-process disk I/O usage.
blktrace – very detailed block layer tracing.
perf / tiptop – device driver profiling.

🧠 When to use

Troubleshooting slow disk I/O.
Detecting I/O starvation or noisy-neighbour workloads.
Analysing LVM/RAID performance issues.

📦 5. Storage & Swap Observability

Tools focusing on physical disks, logical volumes, controllers, and swap usage.

🔧 Tools

iostat – read/write performance.
iotop – which processes are causing I/O.
blktrace – kernel-level I/O event tracing.
swapon -s – view swap devices and utilisation.

🧠 When to use

Swap thrash detection.
Disk queue depth analysis.
Understanding storage behaviour under load.

🌐 6. Network Stack & NIC Observability

These tools examine network interfaces, Ethernet drivers, ports, and NIC statistics.

🔧 Tools

tcpdump – packet-level visibility.
ss / netstat – connections and sockets.
iptraf – per-interface traffic charts.
ethtool – NIC driver settings and link state.
nicstat – interface utilisation.
lldptool – LLDP neighbour discovery.
snmpget – SNMP-based network metrics.

🧠 When to use

Packet drops, retransmits, or MTU mismatches.
NIC offload tuning (TSO, GRO, etc.).
Link speed/duplex mismatch troubleshooting.

🖥️ 7. Hardware Observability (CPU, RAM, Buses)

These tools provide insights into how the hardware itself behaves — including CPU frequency, power states, performance counters, NUMA locality, memory pressure, cache behaviour, and bus throughput.

🔧 CPU Tools

mpstat – Reports CPU usage per core, showing utilisation, steal time, IRQ time, and more.
top – Real-time process monitoring with CPU, load average, and per-thread breakdowns.
ps – Snapshot of process states, CPU usage, memory usage, and scheduling information.
pidstat – Per-thread and per-process CPU utilisation, context switching, and scheduling metrics.
perf – Hardware performance counter profiler (cycles, cache misses, branch mispredictions).
turbostat – Intel-specific tool showing CPU frequencies, C-states, P-states, and turbo boost behaviour.
rdmsr – Reads CPU model-specific registers (MSRs) for extremely low-level introspection.

🔧 Memory Tools

vmstat – Shows paging, swapping, memory pressure, interrupts, and system-wide throughput.
free – Reports total, used, cached, and available system memory.
slabtop – Displays kernel slab allocator statistics (caches, objects, memory used).
numastat – NUMA locality, node memory distribution, and remote memory access counts.
perf (memory events) – Analyses hardware counters related to RAM, cache, and memory bus traffic.

🧠 When to use

NUMA locality and cross-node memory access debugging.
CPU throttling, frequency scaling, or thermal throttling investigations.
Memory pressure analysis, leaking workloads, or kernel slab issues.
High-performance tuning for compute-heavy or latency-sensitive workloads.

📊 8. System-Wide Observability Tools

These tools cover multiple layers at once.

🔧 Tools

sar – historic performance logs across CPU, memory, I/O, network.
dstat – live multi-metric system aggregation.
sysdig – holistic tracing across syscalls, network, containers.
/proc – raw kernel data for metrics, states, drivers, and interfaces.

🧠 When to use

Incident response and baselining.
Long-term trending and anomaly detection.
System-wide correlation across resources.

🛠️ Practical Use Cases

✔ Root Cause Analysis (RCA)

Identify if a slowdown is CPU, memory, network, or storage related.
Trace a misbehaving process through syscalls into the kernel.
Compare observed performance against baseline.

✔ Performance Tuning

Scheduler tracing for latency-sensitive workloads.
NIC tuning via ethtool for high-throughput environments.
Storage insight for LVM/RAID/SSD/HDD tuning.

✔ DevSecOps / Security

eBPF tools for detecting suspicious syscalls.
lsof for auditing unexpected open sockets/files.
sysdig rules for behavioural anomaly detection.

A secure system is one that is understood, not just hardened.

🔐 Observability in DevSecOps

Observability is not just operational — it is security-critical:

Detect unusual syscall patterns (possible intrusion).
Identify crypto miners via CPU and scheduler patterns.
Spot exfiltration via abnormal NIC or TCP behaviour.
Validate hardening changes improve performance rather than degrade it.

📚 References

Brendan Gregg — Linux Performance Tools
Kernel documentation — https://www.kernel.org/doc/
Sysdig, LTTng, SystemTap official docs
eBPF / bpftrace reference guides

Ceph Public Network Migration (No Downtime)

Zepher Ashe — Sun, 08 Mar 2026 00:43:10 +0000

Ceph Public Network Migration (Proxmox)

172.16.0.0/16 → 10.50.0.0/24

No service downtime, no data loss

📌 Context

This procedure documents a live Ceph public network migration performed on a Proxmox-backed Ceph cluster.

The goal was to eliminate management-network congestion while maintaining cluster availability and data integrity.

🎯 Objective
🧱 Key Concepts (Read Once)
🚨 Troubleshooting
⚠️ Risks Considered
✅ Final State

🎯 Objective

Migrate all Ceph traffic (MON, MGR, MDS, OSD front + back) from a congested management network to a dedicated Ceph fabric (e.g. 2.5 GbE switch), while keeping the cluster healthy and online.

🧱 Key Concepts (Read Once)

`public_network`

Client ↔ OSD traffic
MON / MGR control plane
CephFS metadata traffic

`cluster_network`

OSD ↔ OSD replication & recovery (data plane)

Important behaviours

MON & MGR enforce address validation
OSDs bind addresses at restart
/etc/pve/ceph.conf is not authoritative alone — Ceph also uses its internal config database

1️⃣ Prepare the New Ceph Network

Create a dedicated bridge on each node (example: vmbr-ceph):

vim /etc/network/interfaces

auto vmbr-ceph
iface vmbr-ceph inet static
    address 10.50.0.20/24
    bridge-ports eno2
    bridge-stp off
    bridge-fd 0
# Ceph (Fabric)

Assign IPs on the new subnet:

pve2 → 10.50.0.20/24
pve3 → 10.50.0.30/24
pve4 → 10.50.0.40/24

Ensure this network is isolated (no gateway required).

Verify connectivity

ping 10.50.0.30
iperf3 -s / -c <peer>

2️⃣ Add the New Public Network (Dual-Network Phase)

NOTE: Back up the file first

cp /etc/pve/ceph.conf /etc/pve/ceph.conf.bak

Edit /etc/pve/ceph.conf:

public_network = 10.50.0.0/24, 172.16.0.0/16
cluster_network = 10.50.0.0/24, 172.16.0.0/16

⚠️ Do NOT remove the old network yet

Confirm:

Proxmox UI → Ceph → Nodes
ceph config dump

3️⃣ Recreate MONs (One by One)

MONs enforce network validation.

For each node:

pveceph mon destroy <node>
pveceph mon create
ceph -s

✔ Ensure quorum after each step.

4️⃣ Recreate MGRs (One by One)

Recreate standby managers first
Leave the active manager for last

pveceph mgr destroy <node>
pveceph mgr create

Verify:

ceph mgr dump

🔧 Recovery Tip

If a manager fails to start:

systemctl reset-failed ceph-mgr@<node>
systemctl start ceph-mgr@<node>

5️⃣ Recreate CephFS Metadata Servers (MDS)

MDS binds its address at creation time

pveceph mds destroy <node>
pveceph mds create

✔ Verify CephFS health before proceeding.

6️⃣ Remove the Old Public Network

Edit /etc/pve/ceph.conf and remove 172.16.0.0/16:

public_network = 10.50.0.0/24
cluster_network = 10.50.0.0/24

7️⃣ Recreate MONs, MGRs, and MDS (Again)

This ensures all control-plane daemons bind exclusively to the new network.

Order:

MONs (one by one)
MGRs (standbys first, active last)
MDS (one by one)

8️⃣ Protect the Cluster Before Touching OSDs

ceph osd set noout

9️⃣ Restart OSDs (Data Plane Migration)

Restart one OSD at a time:

systemctl restart ceph-osd@<id>
ceph -s

Wait for:

PGs: active+clean

Repeat for all OSDs.

🔟 Remove Protection

ceph osd unset noout

🔎 Verification (Critical)

1️⃣ Verify Ceph daemon addresses

ceph osd metadata <id> | egrep 'front_addr|back_addr'

Expected:

✅ front_addr → 10.50.0.x
✅ back_addr → 10.50.0.x
❌ No 172.16.x.x

2️⃣ Verify traffic is using the Ceph fabric

While Ceph is under load:

ip -s link show vmbr-ceph

RX/TX counters should increase, confirming traffic is not using the management network.

3️⃣ Verify raw network performance (iperf3)

⚠️ Important: iperf3 must be installed on all Ceph nodes to test the fabric correctly.

apt install iperf3

Correct testing method:

Server on one node:

iperf3 -s

Client on a different node:

iperf3 -c <peer_ip> -P 4

Expected for 2.5 GbE Ceph fabric:

~2.1–2.4 Gbit/s
Minimal or zero retransmits
Stable throughput across multiple streams

🚨 Troubleshooting: “OSDs Not Reachable / Wrong Subnet”

Symptom

osd.X's public address is not in '172.16.x.x/16' subnet

Cause

Ceph config DB or MON/MGR cache still references the old network.

Fix (Critical)

Restart ALL MONs (mandatory)

systemctl restart ceph-mon@pve2
systemctl restart ceph-mon@pve3
systemctl restart ceph-mon@pve4

Restart ALL MGRs (mandatory)

systemctl restart ceph-mgr@pve2
systemctl restart ceph-mgr@pve3
systemctl restart ceph-mgr@pve4

(Optional) Clean config DB

ceph config rm global public_network
ceph config rm global cluster_network
ceph config set global public_network 10.50.0.0/24
ceph config set global cluster_network 10.50.0.0/24

Restart OSDs again (one by one).

✔ This should resolve any “OSDs missing / wrong subnet” cases.

⚠️ Risks Considered

Why this change is risky

Changing Ceph cluster networking affects quorum, OSD availability, replication traffic, and client IO. Incorrect sequencing can cause data unavailability or permanent loss.

Failure modes considered

MON quorum loss
OSD flapping
Client IO stalls
Backfill storms
Split-brain conditions

Assumptions

Single Ceph cluster
Dedicated replication network (fabric)
Change executed during low IO window

✅ Final State

Dedicated Ceph fabric (2.5 GbE)
No Ceph traffic on management NIC
MON / MGR / MDS / OSD fully migrated
No data loss
Stable cluster

🙏 Acknowledgements

This migration approach was heavily informed by the following Proxmox forum discussion, which proved critical in resolving address-binding and daemon recreation issues during the Ceph public network transition:

Proxmox Forum – “Ceph: changing public network” https://forum.proxmox.com/threads/ceph-changing-public-network.119116/

In particular, the guidance around:

Temporarily running dual public networks
Recreating MON, MGR, and MDS daemons to force address rebinding
Avoiding full cluster downtime during network migration

was instrumental in achieving a clean, no–data-loss migration.

Many thanks to the contributors in that thread for sharing real-world operational experience.

Why Online Disk Expansion Is Safe on Linux (SAN, LVM2, XFS/Ext4)

Zepher Ashe — Sun, 08 Mar 2026 00:41:02 +0000

Why Online Growth Is Safe for SAN (Fibre Channel), LVM2, and File System (XFS)

Modern enterprise Linux storage stacks are designed for non-disruptive, online capacity expansion, even while filesystems are mounted and under active I/O.

This document explains why each layer—SAN/FC, LVM2, and XFS—fully supports this behaviour.

ME4 LUN → dm-multipath → LVM2 (PV/VG/LV) → XFS/ext4

1. SAN / Fibre Channel (FC) – Non-Disruptive LUN Expansion

Modern enterprise SAN systems (such as Dell EMC ME4) support live, nondisruptive LUN expansion:

Designed for nondisruptive expansion; brief I/O stalls may occur.
No unmount or downtime
Host simply rescans the SCSI bus
Multipath handles updated path geometry automatically

Dell’s ME4 Linux best-practices guide states:

“Resize tasks can be done online without disrupting the applications.”

Source:

https://dl.dell.com/manuals/common/powervault-me4-and-linux-best-practices_en-us.pdf

After expanding the LUN, the host only needs a rescan:

rescan-scsi-bus.sh --resize

multipathd -k
# > multipathd> "resize map mpathX"

Conclusion:

SAN LUN expansion is explicitly engineered to occur online, with no interruption to servers or I/O.

2. LVM2 – Online PV and LV Expansion (Safe While Mounted & Under I/O)

⚠️ Do not proceed unless all paths and the multipath device report the new size.

Ensure the following report ALL paths are resized:
multipath -ll mpathX
blockdev --getsize64 /dev/sdX
blockdev --getsize64 /dev/mapper/mpathX

2.1 PV Resize (`pvresize`) – Safe While Mounted

According to Red Hat’s LVM developers:

“pvresize simply updates the metadata to make LVM aware of the new size.”

“The data area does not change; only the PV extent map is updated.”

— Jonathan Brassow, Red Hat LVM developer

Source:

https://www.redhat.com/archives/linux-lvm/2009-May/msg00040.html

Because it modifies only metadata, pvresize:

Does not touch data blocks
Does not affect the filesystem
Is safe under ongoing I/O
- Provided the block device has been correctly resized on ALL paths
Is routinely run on root filesystems, which cannot be unmounted

Cloud vendor documentation confirming online PV resizing

All three major cloud providers document pvresize as part of their “expand disk without downtime” workflow:

AWS – Expand EBS volumes https://docs.aws.amazon.com/ebs/latest/userguide/recognize-expanded-volume-linux.html
Google Cloud – Resize persistent disks https://cloud.google.com/compute/docs/disks/resize-persistent-disk
Azure – Expand disks *without downtime* https://learn.microsoft.com/en-us/azure/virtual-machines/linux/expand-disks?tabs=ubuntu#expand-without-downtime

These platforms are extremely conservative; documenting pvresize online means it is fully safe and supported.

Conclusion:

pvresize is a safe, online, non-disruptive operation on modern LVM2.

2.2 LV Resize (`lvextend`) – Online and Atomic

lvextend updates:

LVM metadata
device-mapper mappings

These operations are:

Atomic
Metadata is committed atomically via device-mapper table swap
Safe during active I/O
Non-disruptive to mounted filesystems

Cloud vendors (AWS, GCP, Azure) all document lvextend as online, immediately after pvresize.

Conclusion:

LVM2 PV and LV can be grown online, even during live writes, with no unmount required.

3. Online Filesystem Growth

After expanding the block device (SAN → PV → LV), the filesystem must grow to use the new space.

Modern Linux filesystems support online, mounted filesystem expansion.

Filesystem	Online Grow	Notes
XFS v4+	✔	Designed for online growth; cannot shrink
ext4	✔	`resize2fs` supports online grow

3.1 XFS – Designed for Online Filesystem Expansion

Red Hat’s official XFS documentation states:

“The filesystem must be mounted to be grown.”

Source:

https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/storage_administration_guide/xfsgrow

This is one of the clearest vendor statements that XFS online growth is not just supported, but required while mounted.

Why XFS online growth is safe:

Journaled metadata operations
Allocation groups expand in place
Grows only while mounted (required)
Designed for SAN arrays, RAID, HPC
Handles heavy concurrent I/O

Conclusion:

XFS is one of the safest and most robust filesystems for online, mounted, high-throughput growth.

4. Network Share Clients Automatically See Grown Filesystems

When an XFS or ext4 filesystem exported over NFSv4 is grown on the server, clients will automatically reflect the new size without requiring remounts.

NFSv4 is stateful and revalidates filesystem attributes, so the updated capacity becomes visible to clients transparently.

Example on the client:

df -h /mnt/nfs_share

🔴 Older Requirements (Why Some Admins Avoid Online Resize)

Before kernel 2.6.31 and early LVM2/LVM1:

pvresize sometimes failed to detect new sizes
Multipath resizing was unreliable
PV resizing sometimes required pvcreate --restorefile
ext2/ext3 could corrupt if device size changed underneath
SAN rescan tools were buggy
UNIX systems generally required unmounting for geometry changes
LVM1 had no reliable online extension

This produced the legacy rule:

“Never resize storage while mounted.”

Modern RHEL9 (2021+) systems:

Online PV expansion = safe
Online LV expansion = safe
Online XFS growth = officially supported
SAN growth = nondisruptive

The entire modern stack is designed for online operation.

Summary Table

Layer	Operation	Safe While Mounted / Under I/O?	Why
SAN / FC	Expand LUN	✔ Yes	Engineered for nondisruptive growth
LVM2 PV	`pvresize`	✔ Yes	Only metadata updated; no data block changes
LVM2 LV	`lvextend`	✔ Yes	Atomic metadata update; safe under I/O
Filesystems	Online growth	✔ Yes	XFS/ext4 support mounted expansion

DEV Community: Zepher Ashe

Stop Writing Python Like JavaScript: Common Language-Switching Mistakes

Overview

1. Respect Language Idioms

2. Mind the Type System

3. Handle Errors the Right Way

4. Manage Dependencies Correctly

5. Be Conscious of Execution Context

6. Security Practices Don’t Always Translate

7. Testing & Tooling

Linux Observability Tools — A Practical Guide

Linux Observability Tools

Table of Contents

📊 Overview Diagram

🧱 1. Application & User-Space Observability

🔧 Tools

🧠 When to use

🧩 2. System Libraries & Syscall Interface

🔧 Tools

🧠 When to use

🧬 3. Kernel Subsystems Observability

🔧 Tools

🧠 When to use

🔩 4. Device Drivers & Block Layer Observability

🔧 Tools

🧠 When to use

📦 5. Storage & Swap Observability

🔧 Tools

🧠 When to use

🌐 6. Network Stack & NIC Observability

🔧 Tools

🧠 When to use

🖥️ 7. Hardware Observability (CPU, RAM, Buses)

🔧 CPU Tools

🔧 Memory Tools

🧠 When to use

📊 8. System-Wide Observability Tools

🔧 Tools

🧠 When to use

🛠️ Practical Use Cases

✔ Root Cause Analysis (RCA)

✔ Performance Tuning

✔ DevSecOps / Security

🔐 Observability in DevSecOps

📚 References

Ceph Public Network Migration (No Downtime)

Ceph Public Network Migration (Proxmox)

📌 Context

🎯 Objective

🧱 Key Concepts (Read Once)

public_network

cluster_network

Important behaviours

1️⃣ Prepare the New Ceph Network

Verify connectivity

2️⃣ Add the New Public Network (Dual-Network Phase)

3️⃣ Recreate MONs (One by One)

4️⃣ Recreate MGRs (One by One)

🔧 Recovery Tip

5️⃣ Recreate CephFS Metadata Servers (MDS)

6️⃣ Remove the Old Public Network

7️⃣ Recreate MONs, MGRs, and MDS (Again)

8️⃣ Protect the Cluster Before Touching OSDs

9️⃣ Restart OSDs (Data Plane Migration)

🔟 Remove Protection

🔎 Verification (Critical)

1️⃣ Verify Ceph daemon addresses

2️⃣ Verify traffic is using the Ceph fabric

3️⃣ Verify raw network performance (iperf3)

🚨 Troubleshooting: “OSDs Not Reachable / Wrong Subnet”

Symptom

Cause

Fix (Critical)

Restart ALL MONs (mandatory)

Restart ALL MGRs (mandatory)

(Optional) Clean config DB

⚠️ Risks Considered

Why this change is risky

Failure modes considered

Assumptions

`public_network`

`cluster_network`

2.1 PV Resize (`pvresize`) – Safe While Mounted

2.2 LV Resize (`lvextend`) – Online and Atomic