DEV Community

Khalif AL Mahmud
Khalif AL Mahmud

Posted on

Lab Task 11 – Monitoring HTTP Traffic with Httpry on Kali Linux — Passive Capture, Live Monitoring & CLF Conversion

#httpry #kalilinux #livemonitoring #passivecapture

When you start digging into network traffic analysis, you quickly realize that Full Packet Capture tools like Wireshark or tcpdump give you everything — which is often too much. Sometimes you just need the HTTP layer: what requests are being made, to which hosts, using which methods, and when.

That is exactly where Httpry fits in. It is a lightweight, specialized HTTP sniffer that sits quietly on a network interface or reads from a captured pcap file and extracts only what matters — HTTP request and response metadata. No noise, no massive binary dumps, just clean HTTP session data.

In this post I will walk through the full Httpry workflow: building it from source, running passive analysis on a captured pcap file, creating log files, converting those logs to Common Log Format, running live monitoring while generating real HTTP traffic, and then repeating the same CLF conversion on the live capture output.

Why Httpry Instead of Just Wireshark?

Full packet capture has its place, but for HTTP-focused monitoring it comes with overhead — storage costs, parsing time, and complexity. Httpry gives you Packet String Data (PSTR): a selected, human-readable subset of network traffic focused on application-layer protocol strings like hostnames, URLs, HTTP methods, and status codes.

Think of it as the difference between recording an entire phone conversation and keeping only a structured call log. The log is smaller, faster to query, and still tells you who called whom, when, and for how long.

What You Will Need

  • Kali Linux (or any Debian/Ubuntu-based system)
  • A .pcap file from a previous traffic capture session (I used one from Lab 10 generated with daemonlogger)
  • git, gcc, make, libpcap-dev installed
  • Root/sudo access

Step 1 — Update the System

Before anything else, make sure your package lists are current.

sudo apt-get update
Enter fullscreen mode Exit fullscreen mode

Step 2 — Install Build Dependencies

Httpry is not in the default Kali repositories as a pre-built package, so you will compile it from source. It needs libpcap-dev, make, and gcc.

sudo apt install libpcap-dev make gcc
Enter fullscreen mode Exit fullscreen mode

Step 3 — Clone and Build Httpry 

git clone https://github.com/jbittel/httpry.git
cd httpry
make
sudo make install
Enter fullscreen mode Exit fullscreen mode

The make step compiles the source and places the binary in /usr/sbin/httpry. You might see a deprecation warning about pcap_lookupdev — that is harmless on modern libpcap versions.

Verify the install:

httpry --version
which httpry
Enter fullscreen mode Exit fullscreen mode

Step 4 — Locate the Captured pcap File (Part i)

For passive analysis I used a .pcap file captured during a previous lab session with daemonlogger. It was stored at /var/log/daemonlogger/.

cd /var/log/daemonlogger
ls
ls -lh capture.1777551302.pcap
Enter fullscreen mode Exit fullscreen mode

Step 5 — Passive Monitoring on the pcap File (Part ii)

The -r flag tells Httpry to read from a pcap file instead of a live interface. This is called passive monitoring — no active capture, just replaying already-captured traffic.

sudo httpy -r capture.1777551302.pcap
Enter fullscreen mode Exit fullscreen mode

(Replace capture.1777551302.pcap with your actual filename.)

Httpry parses the file and prints every HTTP transaction it finds. Each line shows:

Field Example
Timestamp 2026-04-30 05:21:20.856
Source IP 10.0.2.15
Destination IP 54.39.128.230
Direction > (request) or < (response)
Method GET
Host http.kali.org
URI /kali/pool/main/l/...
HTTP Version HTTP/1.1
Status Code 200 OK / 302 Found

At the end of the output you will see a line like 25 http packets parsed.

Step 6 — Create a Log File and Inspect It (Part iii)

The -o flag writes Httpry output to a file instead of (or in addition to) stdout.

sudo httpry -r capture.1777551302.pcap -o httpry_log.txt
cat httpry_log.txt
Enter fullscreen mode Exit fullscreen mode

The saved file includes a header block showing the Httpry version, copyright, and the field names:

# httpry version 0.1.8
# Fields: timestamp,source-ip,dest-ip,direction,method,host,request-uri,http-version,status-code,reason-phrase
Enter fullscreen mode Exit fullscreen mode

Followed by tab-separated data rows for every HTTP event.

Step 7 — Convert to Common Log Format (Part iv)

Httpry's native log format is useful but not the same as Common Log Format (CLF) — the standard format used by Apache, Nginx, and most web server logs. CLF looks like this:

192.168.1.100 - john [16/Oct/2024:12:15:45 -0700] "GET /index.html HTTP/1.1" 200 1024
Enter fullscreen mode Exit fullscreen mode

To convert, I wrote a small Perl script (log2clf.pl) that reads Httpry's tab-separated output and reformats it:

cat > ~/log2clf.pl << 'EOF'
#!/usr/bin/perl
use strict;
use warnings;

while (<STDIN>) {
    chomp;
    next if /^#/;
    next if /^\s*$/;

    my @fields = split(/\t/, $_);
    next unless @fields >= 7;

    my ($timestamp, $direction, $src_ip, $dst_ip, $method, $host, $path, $proto) = @fields;

    next unless defined $method && $method =~ /GET|POST|HEAD/;

    my $url = "http://" . ($host // "-") . ($path // "/");
    my $time_fmt = $timestamp;

    print "$src_ip - - [$time_fmt] \"$method $url $proto\" - -\n";
}
EOF
Enter fullscreen mode Exit fullscreen mode

Then pipe the Httpry log through it:

cat ~/httpry_log.txt | perl ~/log2clf.pl > ~/httpry_clf.txt
cat ~/httpry_clf.txt
Enter fullscreen mode Exit fullscreen mode

The output now matches CLF structure — IP address, timestamp in brackets, quoted request line, and placeholder dashes for status/bytes (since Httpry does not always capture full response metadata for every entry).

Step 8 — Live Monitoring While Generating HTTP Traffic (Part v)

Now for real-time capture. Run Httpry on your active network interface (eth0 or eth1 depending on your setup) and simultaneously generate HTTP traffic from another terminal.

Terminal 1 — start Httpry:

sudo httpry -i eth0
Enter fullscreen mode Exit fullscreen mode

Terminal 2 — generate HTTP traffic:

curl http://example.com
curl http://google.com
Enter fullscreen mode Exit fullscreen mode

Httpry starts printing HTTP events in real time as the curl requests go out and responses come back. When you are done, press Ctrl+C to stop. It will print a summary like:

^CCaught SIGINT, shutting down ...
48 packets received, 0 packets dropped, 4 http packets parsed
8.4 packets/min, 0.7 http packets/min
Enter fullscreen mode Exit fullscreen mode

Step 9 — Save Live Capture to a File (Part vi)

Repeat the live monitoring but this time save the output to a file using -o:

sudo httpry -i eth0 -o live_capture.txt
Enter fullscreen mode Exit fullscreen mode

Generate some HTTP traffic again with curl, then Ctrl+C to stop. Verify the file was written:

cat live_capture.txt
Enter fullscreen mode Exit fullscreen mode

You will see the same header format as before (# Fields: timestamp,...) followed by the live session data.

Step 10 — Convert the Live Capture to CLF (Part vii)

Apply the same CLF conversion to the live capture file. You can reuse or slightly tweak the Perl script:

cat > ~/log2clf.pl << 'EOF'
#!/usr/bin/perl
while (<STDIN>) {
    chomp;
    next if /^#/;
    next if /^\s*$/;
    my @f = split(/\t/, $_);
    next unless @f >= 7;
    my ($ts, $dir, $src, $dst, $method, $host, $path, $proto) = @f;
    next unless defined $method && $method =~ /GET|POST|HEAD/;
    $path //= "/";
    $proto //= "HTTP/1.1";
    print "$src - - [$ts] \"$method http://$host$path $proto\" - -\n";
}
EOF

cat ~/live_capture.txt | perl ~/log2clf.pl > ~/live_clf.txt
cat ~/live_clf.txt
Enter fullscreen mode Exit fullscreen mode

The output confirms the live traffic converted correctly to CLF:

172.66.147.243 - - [2026-05-04 12:44:34.618] "GET http://example.com/ HTTP/1.1" - -
142.250.187.78 - - [2026-05-04 12:44:38.835] "GET http://google.com/ HTTP/1.1" - -
Enter fullscreen mode Exit fullscreen mode

How to Verify Everything Worked

Check Command Expected Result
Httpry is installed which httpry /usr/sbin/httpry
Passive read works sudo httpry -r file.pcap HTTP lines printed, packets parsed count shown
Log file created cat httpry_log.txt Header + tab-separated HTTP data rows
CLF conversion cat httpry_clf.txt Lines formatted as IP - - [timestamp] "METHOD url proto" - -
Live capture works sudo httpry -i eth0 Real-time HTTP events printed to terminal
Live log saved cat live_capture.txt Same format as passive log with live timestamps
Live CLF done cat live_clf.txt CLF lines from live session

What I Learned

Working through this taught me several things I did not fully appreciate before:

Httpry occupies a specific niche. It is not trying to replace Wireshark or tcpdump. It is purpose-built for extracting HTTP metadata quickly — PSTR (Packet String Data). For network defenders who need to know what HTTP activity happened without storing full packet payloads, this is a practical tool.

Log format matters for toolchain compatibility. Httpry's native format is clean and readable, but most SIEM tools, log parsers, and web analytics tools expect Common Log Format. Writing the Perl conversion script made it clear why standardization exists — a few field reorderings and you can feed Httpry output into tools designed for Apache logs.

Passive vs live monitoring are different operational modes. Reading a pcap file is deterministic and repeatable — useful for forensics and post-incident analysis. Live capture on an interface is real-time but ephemeral. Both have their place, and knowing when to use each is part of being a practical network analyst.

Httpry's -r flag is the key for forensic work. Most forensic workflows start with a pcap you already have. Being able to extract HTTP-layer data from it without re-opening Wireshark is genuinely useful.

Common Mistakes

Mistake Why It Happens Fix
Running Httpry without sudo Packet capture needs root privileges Always use sudo httpry ...
Wrong interface name eth0 vs eth1 vs ens33 varies by system Check with ip a or ifconfig before running
Perl script not filtering # comment lines The Httpry log file starts with comment headers Add next if /^#/; at the start of the while loop
CLF output shows wrong field order Tab-split index off by one Print and count the raw fields to verify @fields alignment
make install fails with "no such file" for man page Kali may not have /usr/man/man1/ pre-created This is a cosmetic error — the binary installs correctly to /usr/sbin/
pcap file has no HTTP traffic Traffic was captured on HTTPS only HTTP (port 80) traffic is needed; HTTPS (443) is encrypted and not visible to Httpry
httpry_log.txt is empty Ran without -q and output went to stdout only Use -o filename.txt to save to file

Conclusion

Httpry is one of those tools that earns its place precisely because it does one thing and does it well. When you need a fast answer to "what HTTP traffic happened between these IPs during this window?" — whether from a historical pcap or a live interface — Httpry gets you there without the overhead of full packet capture.

The CLF conversion step is a practical bridge between Httpry's output and the broader ecosystem of log analysis tools. Once your data is in CLF, you can pipe it into any number of parsers, dashboards, or SIEM inputs.

If you are building a network monitoring toolkit, Httpry deserves a spot alongside your pcap tools — not instead of them, but as a faster, lighter option for HTTP-specific questions.

Top comments (0)