DEV Community

FeiHua
FeiHua

Posted on

JS Reverse Engineering Case Study #4 — A Dynamic Request-Header Signature (HMAC + Custom Character Table + execjs)

#javascript

JS Reverse Engineering Case Study #4 — A Dynamic Request-Header Signature (HMAC + Custom Character Table + execjs)

Part 4 of the "JS Reverse Engineering in Practice — 30 Case Studies" series.
Source code (this case): JS Reverse Engineering — 30 Case Studies · GitHub (Project4)

Note on sanitization: This is a sanitized, educational write-up. Every real domain, endpoint, site-specific constant and cookie name has been replaced with a generic placeholder (example.com, <...>). The custom character table has been reduced to placeholders. No production endpoint or secret is disclosed. The purpose is to explain the technique, not to enable abuse of any specific service.

1. Goal & Background

The target is the company-search API of an enterprise business-information lookup platform (referred to as "Site D"). Its most striking feature is that the signature does not have a fixed name: both the header key and the header value are generated dynamically and may differ on every request, with x-pid being the only thing that ties the two together.

  • Target site: https://www.example.com/
  • Target endpoint: https://www.example.com/<search-endpoint>
  • Parameters to reconstruct: one dynamic request header (neither the header key nor its value is fixed) plus x-pid
  • Difficulty: ⭐⭐⭐⭐ — the algorithm is HMAC, but the key is derived from "the path run through a custom character table" and the value is salted with the page's window.tid, making the chain longer than the previous case.

This project is the archetype of the dynamic request header: the server does not recognise "some fixed header name"; it sends down a pair — a header name that is 20 characters from the middle of one hash, and a value that is a separate HMAC — and validates them together, with x-pid binding the browser session. So the reverse-engineering effort has two jobs: how the header name/value are computed, and where tid / pid come from.

2. Traffic Analysis

Open https://www.example.com/, search for any keyword (say <keyword>), press F12 → Network → Fetch/XHR, and find the search POST.

Look at the Request Headers and you will notice something counter-intuitive — besides the normal headers, there is a header name you have never seen before, and it may change to another name after every refresh:

accept / origin / referer / user-agent ... (normal headers)
x-pid: web_<SESSION_ID>
<20-char lowercase header name>: <a long hex value>
Enter fullscreen mode Exit fullscreen mode

Now look at the Payload:

{"searchKey":"<keyword>","pageIndex":2,"pageSize":20}
Enter fullscreen mode Exit fullscreen mode

The request body is plaintext. The x-pid header is an ID with a fixed prefix, and combined with that "random header name" I concluded:

  • x-pid corresponds to the page's window.pid (a session-level identifier);
  • the "random header name + value" is a signature pair: the name is derived from the endpoint path + the request body, and the value additionally mixes in a window.tid.

📷 [Figure 1 — screenshot suggestion] In the Network panel: the dynamic request header and x-pid of the search request.

Conclusion: what must be reconstructed is the algorithm for the "header name / header value", plus extracting window.pid / window.tid from the page.

3. Locating the Signature Entry Point

First, in the Sources panel, globally search for search to reach the request wrapper; then search for the point where the header is assigned, and you will see it call something like getXXX(...) to generate that dynamic header.

Following it further in, I found the core signing module. I carved this logic out into js/arg.js in the repo. Its structure is: one hash implementation (HMAC-SHA256) + one "path → key" character-table mapping function + two assembly functions:

// From the real repo JS: Project4/<site>/js/arg.js (core module, excerpt)
    AA = {                                   // HMAC over e with key t, returned as a string
        default:function(e, t) { return (0, AA_3.default)(e, t).toString() }
    }

    AA_2 = {
        "default": {
            "n": 20,
            "codes": {                       // a fixed 20-entry table (digits 0-19 -> letters)
                "0": "<c0>", "1": "<c1>", "2": "<c2>", "3": "<c3>", "4": "<c4>",
                "5": "<c5>", "6": "<c6>", "7": "<c7>", "8": "<c8>", "9": "<c9>",
                "10": "<c10>", "11": "<c11>", "12": "<c12>", "13": "<c13>", "14": "<c14>",
                "15": "<c15>", "16": "<c16>", "17": "<c17>", "18": "<c18>", "19": "<c19>"
            }
        }
    }

    AA_3 = {                                 // internally PP.HMAC(SHA-256)
        default:function(e, n) { return new PP.HMAC.init(TT,n).finalize(e) }
    }

    OO = {
        default:function() {
            for (var e = (arguments.length > 0 && void 0 !== arguments[0] ? arguments[0] : "/").toLowerCase(), t = e + e, n = "", i = 0; i < t.length; ++i) {
                var o = t[i].charCodeAt() % AA_2.default.n;
                n += AA_2.default.codes[o]
            }
            return n
        }
    }

    RR = {
        default:function() {
            var e = arguments.length > 1 && void 0 !== arguments[1] ? arguments[1] : {}
                , t = arguments.length > 2 && void 0 !== arguments[2] ? arguments[2] : ""
                , n = (arguments.length > 0 && void 0 !== arguments[0] ? arguments[0] : "/").toLowerCase()
                , i = JSON.stringify(e).toLowerCase();
            return (0,
                AA.default)(n + "pathString" + i + t, (0,
                OO.default)(n))
        }
    }
Enter fullscreen mode Exit fullscreen mode

Once the three functions are clear, everything falls into place:

  • AA.default(e, t): runs HMAC over the string e with key t (AA_3 is internally PP.HMAC.init(TT, n).finalize(e), using a SHA-256 implementation — you can see the SHA-256 initial constant 1779033703 and the round constant 1116352408 in the file);
  • OO.default(path): lowercases path, duplicates it (t = e + e), takes charCodeAt() % 20 for each character, looks the letter up in the codes table, and concatenates a fixed-length key. This "character-table mapping" is what makes the key look random without the server having to send it down;
  • RR.default(path, data, tid): HMAC(path + "pathString" + JSON(body).toLowerCase() + tid, OO(path)).

Finally there is the entry point get_arg exported for Python, which returns the "header name" and "header value" together:

// From the real repo JS: Project4/<site>/js/arg.js (tail)
    TT = "/<search-endpoint>"

    EE = {
        data:{
            "searchKey": name,
            "pageIndex": page,
            "pageSize": 20
        }
    }

    SS = {
        default:function() {
            var list = ["w", "i", "n", "d", "o", "w", ".", "t", "i", "d"];
            return eval(list.join(""))
        }
    }

    oX = {                                   // generates the "header name"
        default:function() {
            var a = AA, o = OO
            var e = arguments.length > 1 && void 0 !== arguments[1] ? arguments[1] : {}
                , t = (arguments.length > 0 && void 0 !== arguments[0] ? arguments[0] : "/").toLowerCase()
                , n = JSON.stringify(e).toLowerCase();
            return (0, a.default)(t + n, (0, o.default)(t)).toLowerCase().substr(8, 20)
        }
    }

    u = (0, RR.default)(TT, EE.data, (0, SS.default)());
    var i = (0, oX.default)(TT, EE.data)

    res = { key:i, value:u }
    return res;
}
Enter fullscreen mode Exit fullscreen mode

The line eval(list.join("")) in SS.default simply spells out window.tid (the author deliberately split it into an array to defeat static search), i.e. the first argument passed into get_arg(WindowTid, ...).

So:

  • header name key = HMAC(path + JSON(body).toLowerCase(), OO(path)).toLowerCase().substr(8, 20) → characters 8-27 of the hash result, i.e. 20 lowercase characters;
  • header value value = HMAC(path + "pathString" + JSON(body).toLowerCase() + window.tid, OO(path)).

The signature of get_arg(WindowTid, name, page) is function get_arg(WindowTid,name,page){ window = global; window.tid = WindowTid; ... }: it first sets window = global and window.tid = WindowTid to patch the environment, so that the later SS.default can obtain window.tid via eval.

📷 [Figure 2 — screenshot suggestion] In the Sources panel: a breakpoint at the oX / RR call site, or the HMAC call stack.

4. Restoring the JS & Emulating the Environment

The only environment patching in this file is a single window = global; almost everything else is pure logic. The real trouble is the prototype-chain issue when "carving the code out" — the author left a very detailed "fix summary" comment at the end of the file, recording five pitfalls:

// From the real repo JS: Project4/<site>/js/arg.js (file-tail fix comment, excerpt)
/*
 Fix summary (5 places): create.toString / AA_4.init.prototype.toString / tox32's i.create;
 plus _doProcessBlock using `for (var t=0;t<80;t++)`, where `var` shadows the parameter t
 (the data array); and mishandling extend and e.init, which permanently rewrites cfg.init,
 making the hash object's high/low both NaN.
 Restored to the original e.$super.init.apply(this, arguments).
 */
Enter fullscreen mode Exit fullscreen mode

What these pitfalls have in common is: when hand-carving crypto-js / a home-made hash, the variable scope and prototype chain of extend, init and _doProcessBlock are so sensitive that the slightest slip makes the hash come out wrong (and it usually does not throw — it just returns the wrong result). Also, in OO.default the character table's n=20 and the codes mapping are hard-coded, and not a single character may be changed when restoring them; path is .toLowerCase()-processed before and after computation, so passing an uppercase path yields a different header name.

On the Python side, all you need to supply is WindowTid (extracted from the page); everything else is computed inside the JS.

5. Python Implementation

First, the "extract pid / tid from the page" part, py/arg/<site>_arg.py: request the search page and parse the inline <script>window.pid='...' with string splitting:

# From the real repo py: Project4/<site>/py/arg/<site>_arg.py
import requests


def site_arg(name:str)-> dict[str,str]:
    headers:dict[str,str] = {
        # ... ordinary browser headers (accept / accept-language / cache-control / dnt /
        # pragma / sec-ch-ua* / sec-fetch-* / user-agent etc.), trimmed here
        "referer": "https://www.example.com/",
        "priority": "u=0, i",
        "upgrade-insecure-requests": "1",
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"
    }
    cookies:dict[str,str] = {
        "<DEVICE_COOKIE>": "<your-device-cookie>",
        "<WAF_COOKIE>": "<your-waf-cookie>",
        "<SESSION_COOKIE>": "<your-session-cookie>",
        "<ANTIBOT_COOKIE>": "<your-antibot-cookie>"
    }
    url:str = "https://www.example.com/web/search"
    params:dict[str,str] = {
        "key": name,
        "p": "2"
    }
    response:requests.Response = requests.get(url, headers=headers, cookies=cookies, params=params)

    result:str = response.text
    WindowPid:str = result.split("<script>window.pid='")[1].split("'")[0]
    WindowTid:str = result.split("<script>window.pid='")[1].split("='")[1].split("';")[0]
    WindowVersion:str = result.split("'</script>")[0].split("window.version='")[1]
    return {"WindowPid":WindowPid, "WindowTid":WindowTid, "WindowVersion":WindowVersion}
Enter fullscreen mode Exit fullscreen mode

Then the main script py/Spider_<site>.py: fetch WindowPid / WindowTid, use execjs to compute the headers, and splice them into the request:

# From the real repo py: Project4/<site>/py/Spider_<site>.py
import requests
import json
from arg.site_arg import site_arg
import os
import execjs
from urllib.parse import quote
from typing import Union,Any

def Spider(Name:str,Page:int)-> None:
    name:str = Name
    pageIn:int = Page
    header_refererNa:str = quote(name,encoding="utf-8")
    header_referer:str = "https://www.example.com/web/search?key=" + header_refererNa + "&p=2" if pageIn == 1 else "https://www.example.com/web/search?key=" + header_refererNa
    arg:dict = site_arg(name)
    WindowPid:str = arg["WindowPid"]
    WindowTid:str = arg["WindowTid"]

    base_path:str = os.path.dirname(os.path.abspath(__file__))
    js_path:str = os.path.join(base_path,"..","js","arg.js")
    js_dir:str = os.path.dirname(js_path)
    os.chdir(js_dir)

    with open(js_path,encoding="utf-8") as f:
        js_code:str = f.read()
    js:Any = execjs.compile(js_code)
    hea_arg:dict[str,str] = js.call("get_arg",WindowTid,name,pageIn)
    key:str = hea_arg["key"]
    value:str = hea_arg["value"]

    headers:dict[str,str] = {
        "Content-Type": "application/json",
         key: value,                          # <- the dynamic header name : value
        # ... ordinary browser headers (accept / origin / referer / sec-ch-ua* / user-agent etc.), trimmed here
        "origin": "https://www.example.com",
        "referer": header_referer,
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
        "x-pid": WindowPid,
        "x-requested-with": "XMLHttpRequest"
    }
    cookie:dict[str,str] = {"<DEVICE_COOKIE>":"<your-device-cookie>", "<WAF_COOKIE>":"<your-waf-cookie>", "<SESSION_COOKIE>":"<your-session-cookie>"}
    url:str = "https://www.example.com/<search-endpoint>"
    Data:dict = {"searchKey": Name, "pageIndex": pageIn, "pageSize": 20}
    data:str = json.dumps(Data, separators=(',', ':'))
    response:requests.Response = requests.post(url, headers=headers, cookies=cookie, data=data)
    print(response.text)
    print(response.status_code)


Spider("<keyword>",2)
Enter fullscreen mode Exit fullscreen mode

Key points:

  • The dynamic header is { key: value } inserted directly into headers (key is that 20-character random name, value is the HMAC result);
  • x-pid uses WindowPid, and referer uses the URL-encoded search page address;
  • pageIndex is passed the Page value (here it is one-based, unlike the previous case);
  • The cookies (<DEVICE_COOKIE> / <WAF_COOKIE> / <SESSION_COOKIE>) are login state, already sanitized to placeholders, and must be replaced.

6. Result

Running python Spider_<site>.py prints the search-result JSON and the status code:

{"status":200,"data":[{"name":"<Company Name>","creditCode":"...","regNo":"...","operName":"..."}, ...],"message":"OK"}
200
Enter fullscreen mode Exit fullscreen mode

A 200 with a company list means the dynamic header was accepted. If you get 401/403 or a risk-control HTML page, it is most likely an expired cookie or an x-pid / tid that does not match the request.

📷 [Figure 3 — screenshot suggestion] The terminal output: the enterprise search results.

7. Pitfalls & Notes

  1. The header name is dynamic and 20 lowercase characters. Do not expect a fixed header name; it is HMAC(...).toLowerCase().substr(8,20). When debugging, print the computed key / value and compare them against the browser Network panel.

  2. window.tid must come from a real page. The first argument of get_arg is the page's window.tid; if you do not use a real value (or use a tid from a stale page), the header value will be wrong. So request the search page once for pid / tid before every run.

  3. The path is case-sensitive. path is .toLowerCase()-processed before hashing, and it must match the internal constant TT = "/<search-endpoint>".

  4. Hand-carving a hash is most dangerous around the prototype chain / scope. The five fix comments at the end of the file (extend mounting, _doProcessBlock's var t shadowing, .cfg.init being permanently rewritten ...) show that once extend / init is handled wrongly, the hash is silently computed wrong. Restore it following the original library structure whenever possible.

  5. Cookies are login state and will expire. <DEVICE_COOKIE> / <WAF_COOKIE> / <SESSION_COOKIE> are sanitized placeholders and must be replaced with your own logged-in values.

  6. pageIndex is one-based, unlike the previous case's zero-based pageNo. Do not confuse the two.

  7. Keep the request rate low and respect the terms of service.

8. Takeaways

  1. "Dynamic header name + dynamic header value" is a subtler layer than a fixed signature: the header name is often just a middle fragment of some hash, bound to the session through x-pid / tid.
  2. An HMAC key is not necessarily sent down in the clear — here the site generates it on the fly with a "path character-table mapping" (charCodeAt() % 20 then a table lookup), a trick worth remembering.
  3. When carving a hash library, the prototype chain and scope are more lethal than the algorithm itself; a wrong result throws no error, so you can only compare it segment by segment against the browser.

Disclaimer: This article is for educational and security-research purposes only. Do not use it for any commercial or illegal activity. All endpoints/sites have been anonymized; always respect the target's terms of service and applicable laws.

Top comments (0)