DEV Community

POTHURAJU JAYAKRISHNA YADAV
POTHURAJU JAYAKRISHNA YADAV

Posted on

From Port 22 to the Culprit: Tracing Outbound SSH on Linux

#linux #devops #security #debugging

A while back I ran into something that made me pause: one of our production EC2 boxes was opening outbound SSH connections to a host nobody could account for. No one had set it up on purpose. Something on the machine, an app, a stray script, maybe a leftover job, was quietly reaching out over port 22.

Noticing the connection was the easy part. The uncomfortable questions came right after:

  • Which process is actually doing this?
  • Which file on disk is that process running from? (Good luck finding it by hand if it's buried under something like /opt/acme/agent/internal/tasks/sync/.)
  • What started it, and what keeps it alive, so I can stop it for good instead of playing whack-a-mole?

The good news: you can answer all of this with standard Linux commands and the /proc filesystem. No agents, no fancy tooling. This post is the walkthrough I wish I'd had that day.

To keep it safe and reproducible, I rebuilt the scenario in a lab, a "mystery" script buried deep under /opt that runs ssh to an unreachable address on a timer. The connection attempt is real; the destination goes nowhere because we explicitly black-hole it with a firewall rule (see the lab setup below). Don't rely on the IP alone to contain the traffic, make the host drop it.

Here's the path we'll follow:

   outbound connection
           │
           ▼
   ss  →  find the PID
           │
           ▼
   /proc/<pid>/exe  →  the binary (often just ssh)
           │
           ▼
   /proc/<pid>/cmdline  →  the full command
           │
           ▼
   parent process  →  the actual script path
           │
           ▼
   pstree  →  the whole ancestry in one view
           │
           ▼
   cgroup  →  owning slice / service (the root cause)
           │
           ▼
   stop it + check how the file got there
Enter fullscreen mode Exit fullscreen mode

Let's work through it.

Reproduce the issue yourself (the lab setup)

Before the investigation, here's how to recreate the "mystery app" on a disposable test VM so you can follow along. It's a script buried under /opt that runs ssh on a loop, started as a plain background process (no systemd).

Run this only on a throwaway lab VM. The target 10.255.255.1 is a private (RFC 1918) address, it is not inherently non-routable, so don't assume the traffic can't leave. To guarantee nothing escapes, add a firewall DROP rule first so the lab host silently black-holes the attempt (this is also what keeps the ssh process hanging in SYN-SENT long enough to inspect by hand):

sudo iptables -A OUTPUT -d 10.255.255.1 -j DROP

Save this as setup_loop.sh:

#!/usr/bin/env bash
# setup_loop.sh — plant a buried script that "phones home" over SSH on a loop.
# Pair with the iptables DROP rule above so the SYN is black-holed and the
# ssh process hangs in SYN-SENT long enough to inspect by hand.
set -euo pipefail

APP_DIR="/opt/acme/agent/internal/tasks/sync"
APP="$APP_DIR/sync_agent.sh"

mkdir -p "$APP_DIR"
cat > "$APP" <<'EOF'
#!/usr/bin/env bash
while true; do
  ssh -o ConnectTimeout=300 -o StrictHostKeyChecking=no \
      -o BatchMode=yes backup@10.255.255.1 "echo sync" >/dev/null 2>&1
  sleep 30
done
EOF
chmod +x "$APP"

# start it as a background loop, no systemd
nohup "$APP" >/dev/null 2>&1 &
disown
echo "Started background loop. Find it with: sudo ss -tnp 'dport = :22'"
Enter fullscreen mode Exit fullscreen mode

Run it:

chmod +x setup_loop.sh
sudo ./setup_loop.sh
Enter fullscreen mode Exit fullscreen mode

That's it, you now have a "rogue" outbound SSH attempt firing in the background, and you can pretend you don't know where it came from. Now let's hunt it down.

Step 1 — Catch the connection and grab the PID

An outbound SSH connection is just a TCP connection to port 22. ss lists sockets, and -p ties each one to the process that owns it:

sudo ss -tnp 'dport = :22'
Enter fullscreen mode Exit fullscreen mode 
State     Recv-Q Send-Q   Local Address:Port       Peer Address:Port     Process
SYN-SENT  0      1        172.31.29.72:44516       10.255.255.1:22       users:(("ssh",pid=1739,fd=3))
Enter fullscreen mode Exit fullscreen mode

The flags, quickly:

  • -t → TCP only
  • -n → numeric, don't waste time resolving names
  • -p → show the owning process
  • dport = :22 → filter to destination port 22

That one line already gives me two things: the destination (10.255.255.1:22) and the PID (1739). Everything else hangs off that PID.

Step 2 — Find the binary behind the PID

Linux exposes a running program's real path at /proc/<pid>/exe, a symlink straight to the file being executed:

sudo readlink -f /proc/1739/exe
Enter fullscreen mode Exit fullscreen mode 
/usr/bin/ssh
Enter fullscreen mode Exit fullscreen mode

Here's a subtlety worth stopping on, because it trips people up:

/proc/<pid>/exe shows the executable, not necessarily the script that matters.

In this case it points at /usr/bin/ssh, which is correct but not useful on its own, ssh is a generic client. The same thing happens with interpreted languages: a Python script shows up as /usr/bin/python3, a Node app as /usr/bin/node. The executable is the interpreter. The actual culprit, the script that decided to run ssh, lives one level up, in the parent process. That's where we're headed, and it's exactly why Step 4 matters.

Step 3 — Read the full command line

/proc/<pid>/cmdline holds the command and its arguments, separated by null bytes. Convert them to spaces to make it readable:

sudo tr '\0' ' ' < /proc/1739/cmdline; echo
Enter fullscreen mode Exit fullscreen mode 
ssh -o ConnectTimeout=300 -o StrictHostKeyChecking=no -o BatchMode=yes backup@10.255.255.1 echo sync
Enter fullscreen mode Exit fullscreen mode

Now I can see what it's trying to do: connect as backup@10.255.255.1 and run echo sync remotely. That's the "what," next I need the "who."

Step 4 — Climb to the parent (where the real answer lives)

No process starts itself. Each one has a parent, and for anything script-driven, the parent is where the real file path shows up (remember: the child's exe was just ssh).

Grab the parent PID and store it in a variable so the next commands stay copy-pasteable:

PPID_VAL=$(ps -o ppid= -p 1739 | tr -d ' ')
echo "$PPID_VAL"
Enter fullscreen mode Exit fullscreen mode

Then inspect the parent the same way:

sudo readlink -f /proc/$PPID_VAL/exe
sudo tr '\0' ' ' < /proc/$PPID_VAL/cmdline; echo
Enter fullscreen mode Exit fullscreen mode 
/usr/bin/bash
bash /opt/acme/agent/internal/tasks/sync/sync_agent.sh
Enter fullscreen mode Exit fullscreen mode

And there's the file I couldn't find by poking around the filesystem:

/opt/acme/agent/internal/tasks/sync/sync_agent.sh
Enter fullscreen mode Exit fullscreen mode

A caveat worth stating plainly: the immediate parent isn't always the real culprit. Real setups often have wrapper processes, a launcher calling a shell calling the actual script, so you may need to walk up several levels before the meaningful file appears. Don't stop at the first parent; follow the chain until you hit something that's clearly the script. The quickest way to see the whole chain at once is pstree (next step):

sudo pstree -aps 1739
Enter fullscreen mode Exit fullscreen mode

This is the single most important step, and it's the reason Step 2's binary (/usr/bin/ssh) was never going to be the full story on its own.

Step 5 — See the whole chain with pstree

Walking parents one PID at a time works, but pstree shows the entire ancestry at once:

sudo pstree -sap 1739
Enter fullscreen mode Exit fullscreen mode
  • -s → include ancestors (parents), not just children
  • -a → show command-line arguments
  • -p → show PIDs

The output you get depends on how the script was launched. Here's what our lab produces, followed by a separate example showing what a systemd-managed version would look like for comparison.

What our lab shows — started by hand inside a session

Because we launched the script manually (through an SSM session, in my case), the tree shows the full launch path:

systemd,1
  └─amazon-ssm-agen,608
      └─ssm-agent-worke,954
          └─ssm-session-wor,977
              └─sh,1002
                  └─sudo,1005 su ubuntu
                      └─bash,1032
                          └─sudo,31465 /opt/acme/agent/internal/tasks/sync/sync_agent.sh
                              └─bash,31466 /opt/acme/agent/internal/tasks/sync/sync_agent.sh
                                  └─ssh,31467 ... backup@10.255.255.1 echo sync
Enter fullscreen mode Exit fullscreen mode

This is where pstree earns its keep. It traced the outbound ssh all the way back through the session manager, sudo, and the login shell to the exact script. In a real incident, that chain quietly answers who was logged in and how the thing got started, not just what it was.

Separate example — if it were a systemd service

This is not what our lab produces, it's here only for comparison. If the same script were installed as a systemd unit instead of launched by hand, the tree would be much shorter:

systemd,1
  └─bash,28101 /opt/acme/agent/internal/tasks/sync/sync_agent.sh
      └─ssh,28622 -o ConnectTimeout=300 ... backup@10.255.255.1 echo sync
Enter fullscreen mode Exit fullscreen mode

Clean and short: the service runs the script, the script runs ssh. Knowing both shapes helps you recognize at a glance whether you're looking at a managed service or something a person started.

Step 6 — Find what keeps it alive (the root cause)

Now ask: who owns this process, and will it come back if I kill it? /proc/<pid>/cgroup tells you which cgroup/slice the process belongs to:

cat /proc/1739/cgroup
Enter fullscreen mode Exit fullscreen mode

In our lab (the script was launched by hand as a background loop, no systemd), you'll see a user/session slice rather than a service, something like:

0::/user.slice/user-1000.slice/session-3.scope
Enter fullscreen mode Exit fullscreen mode

That already tells you something useful: this wasn't a managed service, it was a loose background process someone started in a session. Nothing will respawn it once you kill it.

If instead the script had been installed as a systemd service, the same command would point straight at the unit:

0::/system.slice/acme-sync.service
Enter fullscreen mode Exit fullscreen mode

That single 0::/... line is the cgroup v2 format (most modern distros). On older cgroup v1 systems you'll see several lines prefixed with controller names instead, e.g. 1:name=systemd:/system.slice/acme-sync.service. Either way, look for the .service at the end.

and you'd follow it to the source:

systemctl status acme-sync.service
Enter fullscreen mode Exit fullscreen mode 
● acme-sync.service - ACME Sync Agent
     Loaded: loaded (/etc/systemd/system/acme-sync.service; enabled)
     Active: active (running)
   Main PID: 28101 (bash)
Enter fullscreen mode Exit fullscreen mode

That distinction matters for remediation: a background process is gone once you kill it, but a service will respawn in seconds unless you stop and disable the unit.

Step 7 — Stop it, then figure out how it got there

In our lab it's a background loop, so there's no unit to disable, you kill the process directly. Find the exact PIDs first, then kill those specific PIDs (safer than a broad pkill -f, which can match unrelated processes):

# find the loop and its ssh child, note their PIDs
ps -eo pid,ppid,cmd | grep -E 'sync_agent.sh|backup@' | grep -v grep

# kill the specific PIDs you found (example: 28101 is the bash loop, 1739 the ssh)
sudo kill 28101 1739
Enter fullscreen mode Exit fullscreen mode

If it had been a systemd service instead, you'd stop and disable the unit so it doesn't respawn:

sudo systemctl stop acme-sync.service
sudo systemctl disable acme-sync.service
Enter fullscreen mode Exit fullscreen mode

Either way, two quick forensics questions remain: when did this file appear, and did a package install it (or did someone drop it in by hand)?

stat /opt/acme/agent/internal/tasks/sync/sync_agent.sh
Enter fullscreen mode Exit fullscreen mode

A word of caution on timestamps: stat shows modify (mtime) and change (ctime) times, which tell you when the contents or metadata last changed, not reliably when the file was first installed. Timestamps can also be rewritten (touch, restores, copies), so treat them as a lead, not proof.

For the package check, use the right tool for your distro, this is where a lot of copy-paste blogs get it wrong:

# Debian / Ubuntu
dpkg -S /opt/acme/agent/internal/tasks/sync/sync_agent.sh 2>/dev/null \
  || echo "not owned by any dpkg package"

# RHEL / CentOS / Amazon Linux / Fedora
rpm -qf /opt/acme/agent/internal/tasks/sync/sync_agent.sh 2>/dev/null \
  || echo "not owned by any rpm package"
Enter fullscreen mode Exit fullscreen mode

If the file belongs to no package, that's a strong hint it wasn't part of a legitimate install. Combine that with the stat timestamp (when it landed) and you usually have enough to reconstruct the story, a deploy, a cron job, or something you really didn't want.

The gotcha: connections that die before you can look

Here's the thing that'll bite you in the real world. If the remote end rejects the connection, the ssh process might live only a few milliseconds. You copy the PID, run readlink, and get... nothing:

$ sudo readlink -f /proc/1739/exe
          ← blank, the process already exited
Enter fullscreen mode Exit fullscreen mode

I hit this exact wall. Two simple ways around it:

1. Make it hang long enough to investigate. Point the connection at a black-holed address (a route that silently drops packets, not one that rejects) with a long ConnectTimeout. The process sits in SYN-SENT and gives you time to work by hand. Great for a controlled lab.

2. Freeze the process the instant you catch it:

sudo kill -STOP <pid>   # suspend, do NOT kill
# ... now inspect /proc/<pid>/ at your leisure ...
sudo kill -CONT <pid>   # resume
Enter fullscreen mode Exit fullscreen mode

A word of caution here: SIGSTOP suspends the process mid-flight. Its socket and any resources it was holding stay pinned in whatever state they were in until you resume it. That's completely fine for a controlled investigation on a box you own where you're about to kill it anyway, but don't leave production processes frozen, resume or terminate them deliberately once you've captured what you need.

For this walkthrough, approach 1 is what I actually used: a long ConnectTimeout against a black-holed address keeps the ssh process sitting in SYN-SENT long enough to run every command by hand.

Tear down the lab

When you're done experimenting, remove everything the setup created. Run this only on the disposable lab VM, rm -rf /opt/acme is destructive, so never paste it onto a real host where /opt might hold something you care about.

#!/usr/bin/env bash
# cleanup.sh — remove the lab (LAB VM ONLY)

# 1. Find the lab PIDs and kill only those (avoids broad pkill -f matching)
PIDS=$(pgrep -f '/opt/acme/agent/internal/tasks/sync/sync_agent.sh')
if [ -n "$PIDS" ]; then
  echo "Killing lab PIDs: $PIDS"
  sudo kill $PIDS
fi

# 2. Remove the buried files (lab path only)
sudo rm -rf /opt/acme

# 3. Remove the isolation rule
sudo iptables -D OUTPUT -d 10.255.255.1 -j DROP 2>/dev/null

echo "Cleanup complete."
Enter fullscreen mode Exit fullscreen mode 
chmod +x cleanup.sh
sudo ./cleanup.sh
Enter fullscreen mode Exit fullscreen mode

Cheat sheet

Goal Command
Find outbound SSH + PID sudo ss -tnp 'dport = :22'
Exact binary on disk sudo readlink -f /proc/<pid>/exe
Full command line sudo tr '\0' ' ' < /proc/<pid>/cmdline
Working directory sudo readlink -f /proc/<pid>/cwd
Environment variables sudo tr '\0' '\n' < /proc/<pid>/environ
Open files / sockets sudo ls -l /proc/<pid>/fd
Parent PID ps -o ppid= -p <pid>
Full ancestry tree sudo pstree -sap <pid>
Owning systemd service cat /proc/<pid>/cgroup → systemctl status <unit>
Who installed the file (Debian) dpkg -S <path>
Who installed the file (RHEL) rpm -qf <path>
Freeze a process to inspect it sudo kill -STOP <pid> / kill -CONT <pid>

Heads-up on environ: /proc/<pid>/environ can contain secrets, tokens, DB passwords, API keys. Handy for investigation, but scrub it before pasting output into a ticket, a Slack thread, or a blog post.

Wrapping up

You don't need special tooling to run down a rogue outbound connection. /proc already exposes everything about a live process, its binary, command line, environment, open sockets, and its parents. Pair that with ss to spot the connection and pstree to see the ancestry, and you can go from "something is talking to port 22" to "this exact file, started by this exact service" in a handful of commands.

And none of this is SSH-specific. Swap :22 for :443, :3306, or whatever you're chasing. Only the detection filter changes, the tracing steps are identical.

If you take one thing away: when /proc/<pid>/exe shows a generic binary like ssh, python3, or node, that's not the end of the trail, it's a signal to walk up to the parent. The real answer is almost always one level up.

Top comments (0)