Private Protocol All Garbled in Packet Capture? TraceEagle Custom Protocol Decoding Tutorial

Custom Protocol Decoding

Captured a private / in-house socket protocol, and none of the built-in formats recognize it—data view shows only a pile of bytes? This article teaches you to write a small script that tells the tool how to read it—split a contiguous byte stream into individual messages, strip protocol headers, decompress when needed, and let the tool automatically recognize the rest as structured data. Saving changes takes effect instantly; no need to recapture. The same connection is immediately re-decoded with the new rules.

1. When to Use It

First try switching views and using auto-detection as described in Data Viewing and Decoding. If a connection is still garbled and no built-in format fits, it is likely running a proprietary / private protocol—that's when a custom decoder comes in. Typical signs:

• What you captured is a long-lived socket connection, not standard HTTP; both directions are binary bytes.
• A common form is length prefix + protobuf / JSON / binary, or it is wrapped in a custom header and compressed once.
• You know the format of these bytes (your own protocol, documented, or reverse-engineered), and just need to write the rules for the tool.

If only a standard format was not recognized, you do not need to write a script; first go back to Data Viewing and Decoding and try auto-detection.

2. Prerequisites

• TraceEagle is installed and started, and you have captured the target traffic as in previous articles (this private connection is already in the request / connection list).
• You know how this protocol frames messages: where a message starts, where the length is written, how many header bytes there are, and whether decompression is needed.
• You know a little basic JavaScript syntax (being able to write if, read integers, and slice bytes is enough; you do not need to be an expert).
• Decoding is performed on already captured data, and you can re-decode as many times as needed—no need to worry that a mistake requires recapturing.

3. Step-by-Step Operations

1. Open Custom Codecs

Open the Custom Codecs editor (the decoder management entry). Here you can create, name, and save a list of decoders, and add, modify, or delete them at any time.

2. Create a Decoder

Click New and give it a name (for example, 'In-house Push Protocol'). A decoder is just a function decode(buf, out)—if you do not want to start from a blank page, click Insert Template and choose the skeleton closest to your protocol; changing a few numbers is enough to run it.

3. Write the Parsing Script

The mental model of decode is simple: the tool feeds you the bytes of this connection and repeatedly asks, 'Starting here, can you cut out one complete message?'—you cut out one and tell it how many bytes were consumed, then it asks again with the remaining bytes until nothing is left.

function decode(buf, out) {
  //Buf: The current remaining byte stream to be parsed; Out: Output collector; Return: The number of bytes consumed this time
}

Remembering three conventions is enough:

• buf is the remaining byte stream. Use buf.byteLength to see how much is left. It is the raw buffer, and you cannot directly use buf[0] to get a byte—use the helper functions below, or wrap it yourself with new DataView(buf) / new Uint8Array(buf).
• out.push(...) pushes out one message; you can push multiple at a time. What you push is a byte segment or string returned by a helper function.
• return the number of consumed bytes: > 0 means one message was cut out and that many bytes were consumed from the beginning; the tool calls you again with the rest. Returning 0 (or not writing return) means 'not enough for a complete message yet'; the tool stops and displays the remaining bytes as-is. If you pushed, you must pair it with a corresponding return, otherwise this push will be discarded.

You do not need to worry about direction: the tool runs your script once each for the send stream and receive stream, and automatically labels the direction of each message cut out. You only handle 'how to cut.'

Helpers directly available in the script (so you do not have to reinvent the wheel):

What you want to do	How to write it
Slice a subsegment	sub(buf, off) / sub(buf, off, len) → returns a byte segment
Read integers (big-endian / little-endian, unsigned)	u8(buf, off)、u16be / u16le、u32be / u32le (buf, off)
Convert to text	hex(buf) (hexadecimal), ascii(buf) (read as text)
Detect / decompress	gzipMagic(buf); gunzip / inflate / unzstd / lz4dtx (buf) (returned as-is if it cannot be decompressed)
Debug print	log(...) / console.log(...) → prints to the 'Debug Output' panel

Standard JavaScript (ArrayBuffer, Uint8Array, DataView, JSON, Math, RegExp, etc.) is also available. To read bytes as text, use ascii(buf) (the sandbox does not have browser-only capabilities such as TextDecoder / fetch / setTimeout).

Here are a few of the most common patterns; just change the numbers:

① Length prefix [4-byte big-endian length][payload]—the most typical private protocol:

function decode(buf, out) {
  if (buf.byteLength < 4) return 0     
  const total = 4 + u32be(buf, 0)        
  if (buf.byteLength < total) return 0 
  out.push(sub(buf, 4, total - 4))      
  return total                           
}

② Frame by newline / delimiter:

function decode(buf, out) {
  const i = ascii(buf).indexOf('\\n')
  if (i < 0) return 0                 
  out.push(sub(buf, 0, i))              
  return i + 1                           
}

③ Process the whole thing at once (for example, decompress the entire packet):

function decode(buf, out) {
  out.push(gzipMagic(buf) ? gunzip(buf) : buf) 
  return buf.byteLength                          
}

④ Dispatch by type field (the header carries a type, and different types are handled differently):

function decode(buf, out) {
  if (buf.byteLength < 4) return 0
  const total = 4 + u32be(buf, 0)
  if (buf.byteLength < total) return 0
  const type = u8(buf, 4)                     
  const body = sub(buf, 5, total - 5)
  out.push(type === 2 ? gunzip(body) : body)   
  return total
}

If you need to remember state across multiple calls (counters, the previous message type, etc.), put the variables in the outer scope of decode—they are preserved across multiple calls for the same stream and automatically reset when the direction changes or decoding restarts.

4. Save; It Takes Effect Immediately

Click Save. The script takes effect immediately; no restart and no recapture are needed. Do not be afraid of breaking it: if the script throws an error or times out, it only means 'this time it did not decode, and you see the raw bytes'; it will not interrupt packet capture, let alone lose data. Feel free to modify it boldly.

5. Apply to Matching Traffic

Go back to the request / connection list, right-click the connection you cannot understand → 'Decode As' → choose the decoder you just wrote. The send and receive of the whole connection will immediately be split and displayed according to your rules.

4. Verification: Garbled Bytes Become Structure

After choosing 'Decode As', look at the result area:

• The originally contiguous byte stream is now split into individual messages, each labeled as coming from send ↑ or receive ↓.
• The payload you pushed after stripping headers / decompressing will be automatically recognized again by the tool—if it contains protobuf / JSON / plist, it will be further decoded into a structured view; use the multi-view comparison in Data Viewing and Decoding to inspect it.
• Tail bytes that were not cut out (for example, the last incomplete half-message) will not be lost; they remain at the end as raw data.

If it is not split correctly the first time, modify the script, save, and click 'Decode As' again; the same connection is immediately re-decoded with the new rules—iterate until satisfied.

5. Troubleshooting / Tips

Symptom	Likely cause	What to do
After clicking 'Decode As', it is still raw bytes	decode returned 0 throughout (no message was cut out), or it threw an exception	Look at the 'Debug Output' panel above the result: errors and timeouts are shown there, marked with ↑ / ↓; fix accordingly
A pushed message is not displayed	After push, it returned 0, so this push was discarded	'Push one message' must be paired with 'return how many bytes it occupied' (return > 0)
buf[0] gives undefined	buf is a raw byte buffer; you cannot get a byte by index	Use helpers such as u8(buf, 0), or wrap it with new Uint8Array(buf) / new DataView(buf)
Want to inspect variables while writing	——	In decode, print any value with log(buf.byteLength) or log(hex(sub(buf, 0, 8))); output goes to the 'Debug Output' panel, much faster than guessing
Messages are split too much / too little	Length calculation is wrong (header length omitted, endianness read backwards)	Verify whether total includes the header length; use u32be for big-endian and u32le for little-endian; do not read them backwards
Still garbled after decompression	Wrong decompression algorithm	Use gunzip for gzip, inflate for zlib/deflate, unzstd for zstd, and lz4dtx for block LZ4; if it cannot be decompressed, it is returned as-is; use gzipMagic to detect first
Script changes have no effect	Not saved, or 'Decode As' was not run again	After saving, click 'Decode As' again; decoding reruns on captured data and can be repeated indefinitely

Next Steps

• How to read and switch views after it is split into structures: see Data Viewing and Decoding
• Have not captured this traffic yet: socket / private protocols often need to be taken from inside the program or at the NIC layer; see Application-Layer Packet Capture and NIC Packet Capture.
• Want to compare two messages field by field: see Request Comparison.
• Want to modify fields and resend after decoding the structure: see Request Construction and Replay.