DEV Community: Efim Rovaev

Using pcap-filter syntax for network filtering

Efim Rovaev — Sat, 18 Mar 2023 18:25:55 +0000

libpcap is a library that provides a high-level interface for capturing network traffic. It was developed as an open source project in the 1990s, and is widely used in a variety of network analysis tools, including Wireshark and tcpdump. One of the key features of libpcap is its filtering syntax, which is based on the Berkeley Packet Filter (BPF) syntax.
In short, BPF is implemented as a virtual machine that executes user-defined byte-code programs within the Linux kernel to filter and process network packets.

BPF syntax allows users to create filters that selectively capture only the network traffic they are interested in, based on various criteria such as source and destination IP addresses, protocol type, port number and packet content. Libpcap supports both packet-level and byte-level filtering, which allows users to create filters based on both the contents and the structure of the packets being captured.

Note that in 2014, an extended version of BPF called eBPF (extended Berkeley Packet Filter) has been developed, which offers even more powerful capabilities - it is not only used for networking, but also for various other purposes, such as tracing system calls, kernel instrumentation, and security monitoring. Its versatility has made it a popular choice among developers and system administrators for a wide range of use cases beyond just network analysis. However, eBPF is a topic for a separate article, here we’ll concentrate on BPF libpcap syntax.

Practical Examples

Showing HTTP traffic

tcpdump -i enp0s3 -A src 10.0.2.15 and dst port http

tcpdump is a powerful command-line packet analyser which can be used to display or record traffic.

Here we are using tcpdump on interface enp0s3 (check yours with ifconfig command) to show the traffic in ASCII (-A flag). We are using BPF syntax to specify that we are interested in packets sourced from IP 10.0.2.15 to any host on port 80 (port http part).

Note that we could also use ngrep to show only packets with a specific content.

ngrep (short for network grep) is command-line packet analyser that supports filtering by packet content using grep-like regex syntax. For instance, the following line will capture a request to http://info.cern.ch/

$ ngrep -W byline cern src net 10.0.2.0/24 and dst port http

###
T 10.0.2.15:50660 -> 188.184.21.108:80 [AP] #3
GET / HTTP/1.1.
Host: info.cern.ch.
User-Agent: Wget/1.21.2.
Accept: */*.
Accept-Encoding: identity.
Connection: Keep-Alive.
.

Displaying network stats for a particular host

You can use iftop to get network stats for a specific host.

iftop is a command-line system monitor that focuses on network stats.

iftop -f "dst host github.com"

Capturing packets with specific headers

You can use the involved syntax to check values in protocol flags and content of the packet. For this you usually use protocol[offset:size] or protocol[offset] syntax. There are also some predefined values (see example below) which shows only end packets of each TCP conversation:

tcpdump -i enp0s3 -X 'tcp[tcpflags] & tcp-fin != 0'

Or try this to get all packets longer than 256 bytes – you can see the structure of IPv4 header - e.g. in wiki:

tcpdump -i enp0s3 -X 'ip[2:2] > 256'

Retro-programming - Commodore64 music engine built in TASS

Efim Rovaev — Thu, 02 Mar 2023 20:03:24 +0000

Intro

Many developers have been trying to recreate computer entertainment technology from the past. This trend is known as retro-programming. One of the popular ideas of this trend is mimicking the tech first used in Commodore 64. This device still holds the official Guinness World Record as the best selling single computer model in history. Commodore 64, also referred to as C64, scored sales about 30 mln units between 1982 and 1994 according to independent estimates. Commodore 64 is an iconic machine, a great contribution to the gaming, digital music and art industries we know today.
One of the breakthrough technologies that made Commodore 64 so popular was its built-in sound chip, the MOS Technology 6581 SID (Sound Interface Device). It was capable of supporting three sound channels. It might look ridiculous now, but at the time it was a very impressive achievement for a relatively affordable home device. Also, this particular engine was much appreciated for its warm and very typical sound signature. Some of the most iconic game soundtracks of all time were created using SID.
This article provides an insight on Commodore 64's sound capabilities and how they were used to create music. We will then learn to build a Commodore 64 app using assembler in TASS.
This project is based on an open-source GitHub repo: https://github.com/r4dx/vicious

The Icon: C64 Music Engine

Introduced in 1982, the C64 was famous, among other features, for its sound capabilities. Its sound hardware consisted of a SID chip that supported three voices with envelope generators and a filter. This feature enabled creating complex soundscapes and music compositions, so this chip quickly became a long-time favourite among musicians and composers. In those days, computer music was primarily created by programming the SID chip directly in machine language. This process was complex and time-consuming. It also required programming skills in addition to music talent. However, the results were often impressive.
You can find more about the famous chip and its capabilities here:

The SID Sound Chip of the Commodore 64: https://www.c64-wiki.com/wiki/SID
An Overview of SID Chip Music Engines on the Commodore 64 https://www.c64-wiki.com/wiki/Music_Engine

Modern day: TASS and its capabilities

64tass is an open-source, multi-pass macro assembler for the 65xx processor series. It is written in highly portable C and is aimed at achieving 100% syntax compatibility with Omicron TAS, C64’s native TurboAssembler. The assembler is compatible with all major 6502 CPU variants, including MOS 6502 and 65C02, and supports features like arbitrary-precision integers, Unicode character strings, built-in linker with section support and various memory models. The software is in the phase of active development and is hosted on SourceForge platform (see link below).
Programming a SID chip using TASS, musicians were able to take full advantage of the C64 sound capabilities and create some interesting pieces of music:

To learn more about the TASS macro assembler, you can check out this link: http://tass64.sourceforge.net/

Getting started

Let us install the basic tools to start our retro-programming experiment:

To install 64tass, you need to download the latest version from the official website (http://tass64.sourceforge.net/) and extract it to a folder on your computer. Once extracted, you can add the folder to your PATH environment variable so that you can run the 64tass assembler from the command line
GTK3VICE Emulator: This is a Commodore 64 emulator that is used to run and test your code. To install GTK3VICE, you need to download the latest version from the official website (https://sourceforge.net/projects/vice-emu/files/releases/binaries/windows/GTK3VICE-3.5-win64.7z/download) and extract it to a folder on your computer. Then, you can add the folder to your PATH environment variable so that you can run the GTK3VICE emulator from the command line.

64tass.exe -a --list=listing.asm --cbm-prg main.asm sid.asm 
if errorlevel 1 exit
x64sc.exe a.out

The make.bat file provided is a simple batch file for compiling and running the code we are writing. Here's a brief explanation:

64tass.exe -a --list=listing.asm --cbm-prg main.asm sid.asm Compiles the main.asm and sid.asm source files into a single Commodore 64 executable file, a.out. The -a option enables the listing output and the --list option specifies the output file name to be listing.asm which is a version where macros are expanded. The --cbm-prg option specifies the output format to be a Commodore 64 program file.
x64sc.exe a.out This starts the GTK3VICE emulator and loads the a.out binary, allowing you to listen to the music you have created. With these tools in place, you are now ready to start retro-programming.

Diving into the code

The core code is defined in the sid.asm. Let us have a better understanding of it.
The program defines several macros making it easier to set up the SID chip's audio channels and filter. For example, the setup macro sets up a voice channel to play a specific waveform with a specific set of attack, decay, sustain, and release values. The macro also accepts an optional argument for the pulse wave duty cycle, which sets the duty cycle of the pulse waveform.

setup .macro voice, waveform, attack, decay, sustain, release, pulse_duty_cycle=0
  pha
  voice_reg = VOICE_REGS[\voice]
  .st voice_reg + PulseWaveDutyCycleLo, #(\pulse_duty_cycle & $00ff)
  .st voice_reg + PulseWaveDutyCycleHi, #(\pulse_duty_cycle>>8 & $000f)
  .set_ctr_reg \voice, \waveform
  .st voice_reg + AttackDecay, #(\attack << 4 | \decay)
  .st voice_reg + SustainRelease, #(\sustain << 4 | \release)
  pla
.endm

Please note the usage of st macros - it is employed to store a value in a memory address.

The setup_filter macro sets up the SID chip's filter, which enables filtering specific frequencies from the audio output. This macro takes arguments for the cutoff frequency, resonance, and voice channel mask, which defines what channels will be affected by the filter.

setup_filter .macro cutOffFreq, resonance, v1=false, v2=false, v3=false, hi_pass=false, lo_pass=false, band_pass=false, mute_v3=false
  .set_cutoff \cutOffFreq
  voice_mask := 0
  .if \v1
    voice_mask |= %00000001
  .endif
  .if \v2
    voice_mask |= %00000010
  .endif
  .if \v3
    voice_mask |= %00000100
  .endif
  .st FilterResonanceAndRouting, #(\resonance << 4 | voice_mask)
  mode := 0
  .if \mute_v3
    mode |= %10000000
  .endif
  .if \hi_pass
    mode |= %01000000
  .endif
  .if \band_pass
    mode |= %00100000
  .endif
  .if \lo_pass
    mode |= %00010000
  .endif
  .st FilterModeVolumeControl, #mode
.endm

The encode macro is used to translate musical sheet notation into a series of numerical values that can be played by the SID chip. It takes as input a musical sheet in the form of a string and a default octave. The musical sheet string is then parsed, with each note being converted into a numerical representation based on the octave and the note name.
The encode macro uses the NOTES dictionary to map note names to numerical values. The NOTES dictionary contains the notes from C to B and x which represents a skip note. The pack macro is used to combine the octave and note index into a single numerical value that can be played by the SID chip.
The encode macro is important because it enables the creation of music using a human-readable format, which can then be easily translated into the numerical format required by the SID chip. It provides a convenient way to describe music without having to manually enter numerical values for each note.

NOTES = {"C":0,"C#":1,"Db":1,"D":2, "D#":3, "Eb":3,"E":4,"F":5,"F#":6,"Gb":6,"G":7,"G#":8,"Ab":8,"A":9,"A#":10,"Bb":10,"B":11,"x":12}
SKIP_NOTE = $ff
encode .macro sheet, default_octave
  DELIMITERS = [" "]
  last := ""
  .for i := 0, i < len(\sheet), i += 1
    sym := \sheet[i]
    .if !(sym in DELIMITERS)
    last ..= sym
    .if i != len(\sheet) - 1
        .continue
    .endif
    .endif
    .if len(last) == 0
    .continue
    .endif
    octave := \default_octave
    note_idx := 0
    .if len(last) <= 3
    .if last in NOTES
        note_idx := NOTES[last]
    .else
        octave := last[0] - '0'
        note_idx := NOTES[last[1:]]
    .endif
    .else
    .error "Note should be defined by <= 3 symbols: " .. repr(last)
    .endif
    .if octave > 7 || octave < 0
    .error "Incorrect octave in note: " .. repr(last)    
    .endif
;   .warn repr(last) .. " = octave: " .. repr(octave) .. " note: " .. repr(note_idx)
    .pack octave, note_idx
    last := ""
  .endfor
.endm

This is used in v1, v2 and v3 macros which represent 3 voices (or different tracks so to say) in SID.

v1 .segment sheet, octave=4
  jmp sid__v1_bytes_end
sid__v1_bytes:    
  .encode \sheet, \octave
sid__v1_bytes_end:
.endm

The play_next_uses_xy macro is a macro that is used to play the next note in a sequence of musical notes. The XY in the macro name refers to the X and Y registers.

Conclusion

This is a brief intro into SID programming and a very primitive macro engine to build your music in human-readable format.
Take a look at the main.asm and try to guess which 80s movie OST is it.

Using FUSE to map network statistics to directories

Efim Rovaev — Thu, 02 Mar 2023 19:44:41 +0000

Intro

FUSE is a great Linux tool for mortals to implement file systems and run them in user space. Let's have some fun with it and implement a simple file system that shows network statistics! We'll use procfs - a filesystem to get system stats.
We will use GO to make it easier to write things.
You can try it right now by downloading the repo, building from sources and mounting the filesystem to a directory like that:

$ git clone https://github.com/r4dx/netstatfs
$ cd netstatfs
$ go build
$ mkdir m
$ ./netstatfs --mount=m&
$ ls m

m folder will contain directories - each representing a process, within each folder there will files - each representing a socket connection.
Now, when we know what we want to achieve, let's take a closer look into tools we'll be using.

FUSE

FUSE, or Filesystem in Userspace, is a software interface that enables users to create their own file systems without writing drivers. To access a file or directory in FUSE, a program first sends a request to the FUSE library that resides in the user space. This library then sends a request of its own to the kernel through the FUSE device file. The kernel forwards the request to the file system implementation in the user space (something that we'll implement!).

This is, of course, a very brief description of FUSE. You can dive into detail as deep as you want (or need) by visiting these resources:

Official FUSE website with detailed information about the technology, its design and the ways it works: https://github.com/libfuse/libfuse
Wikipedia page on FUSE covering its history, design and implementation: https://en.wikipedia.org/wiki/Filesystem_in_Userspace
Linux manual page on FUSE, providing an insight into using the tool in Linux environments: https://man7.org/linux/man-pages/man4/fuse.4.html
There are also several online step-by-step tutorials on FUSE: ** https://www.ibm.com/developerworks/library/l-fuse/index.html ** https://www.cs.nmsu.edu/~pfeiffer/fuse-tutorial/html/index.html

How does netstat do it?

Apart from FUSE, our project employs the proc file system, a Linux virtual file system providing information about the system and processes running on it. We will be using procfs to retrieve network statistics.
One interesting note is that Linux netstat command uses procfs to retrieve network stats. This is how it does it:

It opens the /proc/ directory and reads all numerical IDs there. These IDs stand for the processes that are currently running on the system.
For each process, it reads all IDs in /proc/{processId}/fd/
It resolves the links of socket file IDs, so they are wrapped in the following format: socket:[socketInode] or [0000]:socketInode
Then it stores the mapping of socketInode to its corresponding process in the pgr_hash hash map.
Finally, it reads network protocol files in /proc/net/{tcp,tcp6,udp,udp6,igmp,igmp6,unix} that have an inode field. This field is then used to retrieve the process from the pgr_hash hash map.

It's worth noting that the ss command, which serves a similar purpose, uses the netlink protocol instead. To learn more about this, you can refer to the following link: https://man7.org/linux/man-pages/man7/sock_diag.7.html

Diving into the Code

Now let us take a closer look at the code and discuss it bit by bit:

netstatfs.go implements a file system using the bazil.org/fuse library. The file system exposes information about the network connections of the running processes running on the system.

The main function first sets up the mount point for the file system by parsing a command-line argument. Then, it creates a connection to the FUSE library by calling fuse.Mount. The NewNetstatfs function creates a new instance of the Netstatfs struct, which acts as the root node for the file system. Finally, the function serves the Netstatfs instance as a FUSE file system by calling fs.Serve.

func main() {
    mountpoint := flag.String("mount", "", "mountpoint for FS")
    flag.Parse()
    conn, err := fuse.Mount(*mountpoint, fuse.FSName("netstatfs"))
    if err != nil {
     panic(err)
    }
    defer conn.Close()
    netstatfs, err := NewNetstatfs()
    if err != nil {
     panic(err)
    }
    err = fs.Serve(conn, &netstatfs)
    if err != nil {
     panic(err)
    }
}

Let's take a look at Netstatfs struct:

type Netstatfs struct {
    ProcessProvider ProcessProvider
    SocketProvider  SocketProvider
    FileIdProvider  FileIdProvider
    RootINode       uint64
}

Those are self-explanatory - notice that The FileIdProvider provides unique file IDs for each node in the file system, and the RootINode is the inode number of the root node.

The Root method of Netstatfs returns the root directory of the file system. The root directory is implemented as an instance of the RootDir struct, which has methods to return its attributes and the contents of the directory. The contents of the root directory are names of the processes running on the system, with each name having the format <process ID>_<process name>.

func (me Netstatfs) Root() (fs.Node, error) {
    return RootDir{Root: &me}, nil
}

type RootDir struct {
    Root *Netstatfs
}

Look at the ReadAllDir method of RootDir struct - it uses ProcessProvider to get list of running processes and represent them as directories in our filesystem:

func (me RootDir) ReadDirAll(ctx context.Context) ([]fuse.Dirent, error) {
    processes, err := (*me.Root).ProcessProvider.GetProcesses()
    if err != nil {
        return nil, err
    }
    result := make([]fuse.Dirent, len(processes))
    for i, process := range processes {
        fn := processNameToFileName(
            process.Id, process.Name)
        inode, err := (*me.Root).FileIdProvider.GetByProcessId(process.Id)
        if err != nil {
            return nil, err
        }

        result[i] = fuse.Dirent{Inode: inode,
            Name: fn,
            Type: fuse.DT_Dir}
    }
    return result, nil
}

Each process directory is then implemented as an instance of the ProcessDir struct. The ReadDirAll method returns the network sockets associated with the process. The names of the files have the format <socket local address>:<socket local port>_<socket remote address>:<socket remote port>.

type ProcessDir struct {
    Root    *Netstatfs
    Process Process
    INode   uint64
}

func (me ProcessDir) ReadDirAll(ctx context.Context) ([]fuse.Dirent, error) {
    sockets, err := (*me.Root).SocketProvider.GetSockets(me.Process.Id)
    if err != nil {
        log.Printf("Couldn't get sockets for process=%d: %s\n", me.Process.Id, err)
        return nil, err
    }
    result := make([]fuse.Dirent, len(sockets))
    for i, socket := range sockets {
        inode, err := (*me.Root).FileIdProvider.GetBySocketId(me.Process.Id, socket.Id)
        if err != nil {
            log.Printf("Couldn't get fileid for process=%d, socket=%d: %s", me.Process.Id, socket.Id, err)
            return nil, err
        }
        result[i] = fuse.Dirent{Inode: inode,
            Name: socketToFileName(socket),
            Type: fuse.DT_File}
    }
    return result, nil
}

Notice that ProcessDir is connected to RootDir via implementing the Lookup method.

func (me RootDir) Lookup(ctx context.Context, name string) (fs.Node, error) {
    id, err := fileNameToProcessId(name)
    if err != nil {
        return nil, err
    }
    process, err := me.Root.ProcessProvider.GetProcessById(id)
    if err != nil {
        return nil, err
    }
    inode, err := (*me.Root).FileIdProvider.GetByProcessId(id)
    if err != nil {
        return nil, err
    }
    return ProcessDir{Root: me.Root,
        Process: process,
        INode:   inode}, nil
}

Now, when we've unveiled how the FUSE part works, let's take a quick glimpse on how do we get the list of open sockets for a process.
Basically this is what GetSockets does:

func (me procfsSocketProvider) getProcessSocketInternal(processId uint, socketId uint64) (ProcessSocket, error) {
    file := strconv.FormatUint(uint64(processId), 10) + "/fd/" + strconv.FormatUint(socketId, 10)

    resolved, err := me.procfs.Readlink(file)
    if err != nil {
     return ProcessSocket{}, err
    }
    inode, err := me.getINodeIfSocket(resolved)
    if err != nil {
     return ProcessSocket{}, err
    }
    return ProcessSocket{INode: inode, ProcessId: processId,
     Id: socketId, SocketInfo: SocketInfo{}}, nil
}

func (me procfsSocketProvider) GetSockets(processId uint) ([]ProcessSocket, error) {
    base := strconv.Itoa(int(processId)) + "/fd/"

    files, err := me.procfs.Readdirnames(base)
    if err != nil {
     return nil, err
    }
    result := make([]ProcessSocket, 0)
    for _, file := range files {
     fd, err := strconv.ParseUint(file, 10, 64)
     if err != nil {
         continue
     }
     socket, err := me.getProcessSocketInternal(processId, fd)
     if err != nil {
         continue
     }
     result = append(result, socket)
    }
    err = me.fillSocketInfo(result)
    if err != nil {
     return nil, err
    }

    return result, nil
}
func (me procfsSocketProvider) getINodeIfSocket(src string) (uint64, error) {
    matches := me.socketINodeRe.FindStringSubmatchIndex(src)
    outStr := string(me.socketINodeRe.ExpandString([]byte{}, "$inode", src, matches))
    res, err := strconv.ParseUint(outStr, 10, 64)
    if err != nil {
     return 0, err
    }
    return res, nil
}

The result is a list of all the sockets for the given process with detailed information about each socket.

Conclusion

I hope this helped you get a quick glimpse in what you can do with the power of FUSE. If you'll read the source further - you'll see that with little effort we could even implement reading the content of each "file" (socket in our case) by using PCAP library.

DEV Community: Efim Rovaev

Using pcap-filter syntax for network filtering

Practical Examples

Showing HTTP traffic

Displaying network stats for a particular host

Capturing packets with specific headers

Links

Retro-programming - Commodore64 music engine built in TASS

Intro

The Icon: C64 Music Engine

Modern day: TASS and its capabilities

Getting started

Diving into the code

Conclusion

Using FUSE to map network statistics to directories

Intro

FUSE

How does netstat do it?

Diving into the Code

Conclusion