DEV Community: Rustem Kamalov

Here's how to brew obfuscation in JavaScript without burning the lab: AST, Babel, plugins.

Rustem Kamalov — Fri, 10 Jan 2025 04:00:52 +0000

Introduction

Hey, have you ever thought about how cool and unique your algorithms are? 😎 A lot of programmers and companies do, which is why they might be hesitant to share their work with everyone. This problem gets a little better if part of the code is moved to the server (for client-server applications), but this approach isn't always possible. Sometimes, we have to leave sensitive code sections right out in the open.

In this article, we're going to take a look at obfuscation in the JavaScript, creating ways to hide algorithms and make it harder to study code. We'll also be exploring what AST is and providing tools that can be used to interact with it to implement obfuscation.

What's the deal?

Here's a silly example. Let's imagine this situation:

Bob goes to a site that's giving away computer monitors (here it is -> 📺). Bob's monitor is better, but free stuff is always nice!
When Bob visits the site, JavaScript runs in the browser, collecting data about the user's device and sending it to the server. Let's say this is it:

let w = screen.width, h = screen.height;
// Let's say there's a logic with some check. 
console.info(w, h);

Unfortunately, Bob can't access the giveaway page, and he's pretty upset about it. He doesn't understand why. Then he learns in the rules of the giveaway that users with big, good monitors are not allowed.
Luckily, Bob had taken some computer science classes in high school. He opens the developer console decisively by pressing F12, studies the script, and realizes that the organizers check the screen resolution. He then decides to participate from his phone and successfully passes the test.

A fictional story with a happy ending - but it couldn't have been this good if the main character had seen this instead of the previous code:

l=~[];l={___:++l,$$$$:(![]+"")[l],__$:++l,$_$_:(![]+"")[l],_$_:++l,$_$$:({}+"")[l],$$_$:(l[l]+"")[l],_$$:++l,$$$_:(!""+"")[l],$__:++l,$_$:++l,$$__:({}+"")[l],$$_:++l,$$$:++l,$___:++l,$__$:++l};l.$_=(l.$_=l+"")[l.$_$]+(l._$=l.$_[l.__$])+(l.$$=(l.$+"")[l.__$])+((!l)+"")[l._$$]+(l.__=l.$_[l.$$_])+(l.$=(!""+"")[l.__$])+(l._=(!""+"")[l._$_])+l.$_[l.$_$]+l.__+l._$+l.$;l.$$=l.$+(!""+"")[l._$$]+l.__+l._+l.$+l.$$;l.$=(l.___)[l.$_][l.$_];l.$(l.$(l.$$+"\""+(![]+"")[l._$_]+l.$$$_+l.__+"\\"+l.$__+l.___+"\\"+l.__$+l.$$_+l.$$$+"\\"+l.$__+l.___+"=\\"+l.$__+l.___+"\\"+l.__$+l.$$_+l._$$+l.$$__+"\\"+l.__$+l.$$_+l._$_+l.$$$_+l.$$$_+"\\"+l.__$+l.$_$+l.$$_+".\\"+l.__$+l.$$_+l.$$$+"\\"+l.__$+l.$_$+l.__$+l.$$_$+l.__+"\\"+l.__$+l.$_$+l.___+",\\"+l.$__+l.___+"\\"+l.__$+l.$_$+l.___+"\\"+l.$__+l.___+"=\\"+l.$__+l.___+"\\"+l.__$+l.$$_+l._$$+l.$$__+"\\"+l.__$+l.$$_+l._$_+l.$$$_+l.$$$_+"\\"+l.__$+l.$_$+l.$$_+".\\"+l.__$+l.$_$+l.___+l.$$$_+"\\"+l.__$+l.$_$+l.__$+"\\"+l.__$+l.$__+l.$$$+"\\"+l.__$+l.$_$+l.___+l.__+";\\"+l.__$+l._$_+l.$$__+l._$+"\\"+l.__$+l.$_$+l.$$_+"\\"+l.__$+l.$$_+l._$$+l._$+(![]+"")[l._$_]+l.$$$_+".\\"+l.__$+l.$_$+l.__$+"\\"+l.__$+l.$_$+l.$$_+l.$$$$+l._$+"(\\"+l.__$+l.$$_+l.$$$+",\\"+l.$__+l.___+"\\"+l.__$+l.$_$+l.___+");"+"\"")())();

I assure you, it's not gibberish, it's JavaScript! And it performs the same actions. You can try to run the code in the console here.

I guess in this case, our hero would've just accepted his fate by not taking part in the giveaway, and the organizers would've kept their plan.

So what's the point here? Congrats - you've learned about the jjencode tool and what obfuscation is and what role it can play.

In summary, obfuscation is the process of converting program code or data into a form that's hard for humans to understand but still works for a machine or program.

Hiding secrets. Quick way

Enough theories, let's move on to more practical examples 👨‍🔬. Now let's try to convert the code with the help of obfuscations that you are more likely to find on the Internet. Let's take a more interesting code that contains our “know-how” operations. And it is highly undesirable that everyone who is not too lazy to reach F12 can find out about them:

function getGpuData(){
  let cnv = document.createElement("canvas");
  let ctx = cnv.getContext("webgl");
  const rendererInfo = ctx.getParameter(ctx.RENDERER);
  const vendorInfo = ctx.getParameter(ctx.VENDOR);

  return [rendererInfo, vendorInfo]
}

function getLanguages(){
  return window.navigator.languages;
}

let data = {};
data.gpu = getGpuData();
data.langs = getLanguages();
console.log(JSON.stringify(data))

This code collects device and browser data and outputs the result to the console, for example (we'll use the output as a metric of the code's performance):

{"gpu":["ANGLE (NVIDIA, NVIDIA GeForce GTX 980 Direct3D11 vs_5_0 ps_5_0), or similar","Mozilla"],"langs":["en-US","en"]}

Now let's take the above code and modify it with a popular obfuscator for JS - obfuscator.io. As a result, we will get a code like this:

function _0x9591(_0x587f42,_0x4b4b1a){const _0x581ade=_0x581a();return _0x9591=function(_0x9591f0,_0x20e0a4){_0x9591f0=_0x9591f0-0x18a;let _0x2b716b=_0x581ade[_0x9591f0];return _0x2b716b;},_0x9591(_0x587f42,_0x4b4b1a);}const _0x5d747e=_0x9591;(function(_0x41cdc1,_0x26e305){const _0xd47419=_0x9591,_0x3c47fc=_0x41cdc1();while(!![]){try{const _0x1c6e63=-parseInt(_0xd47419(0x18c))/0x1*(parseInt(_0xd47419(0x18d))/0x2)+parseInt(_0xd47419(0x18f))/0x3+-parseInt(_0xd47419(0x18b))/0x4+-parseInt(_0xd47419(0x195))/0x5+parseInt(_0xd47419(0x196))/0x6+-parseInt(_0xd47419(0x19e))/0x7*(parseInt(_0xd47419(0x192))/0x8)+parseInt(_0xd47419(0x19a))/0x9;if(_0x1c6e63===_0x26e305)break;else _0x3c47fc['push'](_0x3c47fc['shift']());}catch(_0x2210e4){_0x3c47fc['push'](_0x3c47fc['shift']());}}}(_0x581a,0x5b85c));function _0x59d10d(){const _0x12260c=_0x9591;let _0x14403e=document[_0x12260c(0x197)](_0x12260c(0x191)),_0xf297ee=_0x14403e[_0x12260c(0x19b)](_0x12260c(0x199));const _0x16d7eb=_0xf297ee[_0x12260c(0x19f)](_0xf297ee[_0x12260c(0x198)]),_0x3174f4=_0xf297ee[_0x12260c(0x19f)](_0xf297ee[_0x12260c(0x193)]);return[_0x16d7eb,_0x3174f4];}function _0x157cda(){const _0x52b881=_0x9591;return window[_0x52b881(0x19c)][_0x52b881(0x18a)];}let _0x421797={};_0x421797[_0x5d747e(0x19d)]=_0x59d10d(),_0x421797[_0x5d747e(0x194)]=_0x157cda(),console[_0x5d747e(0x190)](JSON[_0x5d747e(0x18e)](_0x421797));function _0x581a(){const _0x3fdf5e=['webgl','15135525QqurjW','getContext','navigator','gpu','304409xUlnUb','getParameter','languages','1546148RYMKQN','14903JFRqxJ','96TioORm','stringify','817929YcOxtF','log','canvas','80ELkOfJ','VENDOR','langs','3339820dAlRZJ','3751338qfcHSk','createElement','RENDERER'];_0x581a=function(){return _0x3fdf5e;};return _0x581a();}

Voila! Now, only a machine will be happy to parse this code (you and I are probably not among them 🤔). Nevertheless, it still works and produces the same result. Note the changes:

Line breaks and extra spaces are gone.
Variable names have been replaced with uninformative ones like _0x587f42.
Strings and object properties have been converted into function calls that return their values from an array by index. For example, document.createElement(“canvas”) turned into document[_0x12260c(0x197)](_0x12260c(0x191)). This was made possible by using computed properties.

The last technique is perhaps the most nasty in this case, in terms of burdening static code analysis.

Alright, looks like all the secrets are hidden. Shall we deploy the code to production?

Wait... If there are services for code obfuscation, perhaps there are some that can pull this stuff back? Absolutely 😏, and more than one! Let's try to use one of them - webcrack. And see if we can get the original, readable code. Below is the result of using this deobfuscator:

function _0x59d10d() {
  let _0x14403e = document.createElement("canvas");
  let _0xf297ee = _0x14403e.getContext("webgl");
  const _0x16d7eb = _0xf297ee.getParameter(_0xf297ee.RENDERER);
  const _0x3174f4 = _0xf297ee.getParameter(_0xf297ee.VENDOR);
  return [_0x16d7eb, _0x3174f4];
}
function _0x157cda() {
  return window.navigator.languages;
}
let _0x421797 = {};
_0x421797.gpu = _0x59d10d();
_0x421797.langs = _0x157cda();
console.log(JSON.stringify(_0x421797));

Oops 🙊. Of course, it did not return the names of variables, but thanks for that.

So it turns out that the only obstacle to calmly studying our code in this case is the researcher's willpower to use deobfuscator. Undoubtedly, it is also possible to use other solutions and customizations, but for any popular obfuscation, we should most likely expect popular deobfuscation.

Should we despair and give up our secrets without a fight? Of course not! Let's see what more we can do....

How you become an obfuscator

An obfuscator - sounds like some kind of mage from a fantasy universe, doesn't it? 🧙🏻‍♂️
Definitely, someone can obfuscate code while writing it and is a born magician. You may even have unintentionally been able to cast such spells yourself for a while. But what should you do now if the skills have disappeared thanks to the criticism of “senior programmers” and you have an idea that potentially allows you to make the program difficult to investigate? In this case, it makes sense to turn to tools that interact with the code structure itself and allow you to modify it. Let's take a look at them.

ASTools

It's also possible to try to modify the code by interacting with it simply as with text, replacing certain constructions with regular expressions and so on. But I would say that following this way, you have more chances to ruin your code and time than to obfuscate it.

For more reliable and controlled modification it makes sense to bring it to an abstract structure, a tree (AST - abstract syntax tree), passing through which we can change the elements and constructs we are interested in.

There are different solutions for working with JS code, with differences in the final AST. In this article, we will use babel for this purpose. You don't need to install anything, you can experiment with everything on such resource as astexplorer.

(If you don't want to mess with babel, check out shift-refactor. It allows you to interact with AST using **CSS selectors. Pretty minimalistic and convenient approach for learning and modifying code. But it uses a specific version of AST, different from babel. You can test your CSS queries for this tool at shift-query interactive demo).

0. Working with AST

Now let's see how these tools can be easily used without leaving the browser, based on a simple example. Suppose we need to change the name of the test variable in the same-named function to changed:

function test(){ // Not here
  let test = "some data"; // Should become changed
  let id = "";
  console.log(test); // Should become changed
}

test(); // Not here

Paste this code into astexplorer (select JavaScript and @babel/parser from above), it should appear as an AST there. You can click on the test variable to see the syntax for this code section in the right window:

To solve our problem, we can write the following babel plugin, which will parse our code and look for all names\identifiers in it and rename them if certain conditions are met. Let's paste it into the bottom left window in astexplorer (turn on the transform slider and select babelv7 to make it appear):

function transformCode() {
  return {
    name: "change-name",
    visitor: {
      Identifier(path) {
        // Output information about the current node and environment
        console.log(path) 

        // We are only interested in the name “test”
        // + it should be in the function
        if(
          path.node.name === "test" && 
          path.parent.type === "FunctionDeclaration"
        ) {
           // Rename. This method will take care of all the references 
          path.scope.rename(path.node.name, "changed")
        }
      },
    },
  };
}

module.exports = transformCode;

Console output is included in this plugin for a reason. This allows us to debug our plugin by examining the output in the browser console. In this case, we output information about all nodes of Identifier type. This information contains data about the node itself (node), the parent node (parent), and the environment (scope- contains variables created in the current context and references to them):

Thus, in the bottom right window, we can notice that the variable in our source code has been successfully changed without affecting other identifiers:

function test() {
  let changed = "some data"; // <-
  let id = "";
  console.log(changed); // <-
}

I hope, based on this example, it became a little clearer how we can parse and modify the code. Anyway, let me summarize the work done:

We converted the code to AST using babel via astexplorer.
By examining the AST, we saw that the test variable is labeled with the Identifier type, the name of which can be defined using the name property.
Next, using our babel plugin, we bypassed all the identifiers and changed the name of those in the function with the name test to changed.

1. Hiding function and variable names

It is clear now how code modification can be done. Let's try something more useful, which we will be able to call obfuscation :) We'll take a more complex code we tried to obfuscate in the previous section. Now we'll change all the names of variables and functions in it to random ones. So, a potential reverse engineer would have less info about the purpose of some code elements.

Also, feel free to use any JS code to debug problems. ~~As they say, there's no better teacher than pain 🥲~~.

The following plugin will help us to get the job done:

// Storage with used identifier names
const usedIdentifiers = new Set();

// Generates a random string from the `characters` alphabet up to 3 chars long
// The generated string must be unique (not previously used by identifiers)
function generateRndName() {
  const characters = "abcdefghijklmnopqrstuvwxyz_";
  let randomIdentifier = "";
  do {
    const length = Math.floor(Math.random() * 4); 
    for (let i = 0; i <= length; i++) {
      randomIdentifier += characters.charAt(
        Math.floor(Math.random() * characters.length)
      );
    }
  } while (usedIdentifiers.has(randomIdentifier));
  usedIdentifiers.add(randomIdentifier);
  return randomIdentifier;
}

// Go through all nodes of `Identifier` type 
// Change their names to random ones along with all references
function transformCode() {
  return {
    name: "hide-names",
    visitor: {
      Identifier(path) {
        path.scope.rename(path.node.name, generateRndName());
      },
    },
  };
}

module.exports = transformCode;

What does this code do? Pretty much the same as in the previous example:

We pass through all AST nodes of Identifier type;
This time, we use the generateRndName function to randomly generate names for identifiers without any conditions;
Generating unique names guarantees that we will not randomly get a duplicate name that may break the program logic.

As a result of our plugin execution, we get the following code with random variables names and functions:

function hjj() {
  let bq = document.createElement("canvas");
  let c = bq.getContext("webgl");
  const a_x = c.getParameter(c.RENDERER);
  const nry = c.getParameter(c.VENDOR);
  return [a_x, nry];
}

function sztc() {
  return window.navigator.languages;
}

let _o = {};
_o.gpu = hjj();
_o.langs = sztc();
console.log(JSON.stringify(_o));

You can check it by executing the code in console - after our manipulations, it still works! And this is the main quality of a good obfuscator ✨.

But what about the quality of our obfuscation? As for me - the evil is not too strong yet: even by replacing the names, it will be easy for an experienced programmer to understand the purpose of this code. And what's the point if any JS minifier can handle this task. Is it possible now to do something more practical and troublesome for a reverser? There is one more spell...

2. Hide 🪄 everything!

I may have been a bit confident when I wrote “everything”, but what we are going to do now will hide the actions of our code to the maximum extent possible. In this section, we will conceal strings and various object properties in order to complicate static analysis and potentially prevent the “client” from digging into our code!

Let's take the code with hidden names obtained at the previous stage and apply the following plugin to it:

function transformCode(babel) {
  const { types: t } = babel;

  // All of our properties/strings to be replaced in the code will be here.
  let data = [];

  return {
    name: "hide-props-strings",
    visitor: {
      //1. Find the `Program` root node
      // And insert into the beginning of the code a function that returns string properties by index
      Program(path) {
        // Body of the created function
        let funcBody = t.blockStatement([
          // Declare the variable that stores the string properties
          // let data = [...]
          t.variableDeclaration("let", [
            t.variableDeclarator(t.identifier("data"), t.arrayExpression(data)),
          ]),

          // return data before decoding it from base64
          // return atob(data[data_index])
          t.returnStatement(
            t.callExpression(t.identifier("atob"), [
              t.memberExpression(
                t.identifier("data"),
                t.identifier("data_index"),
                true
              ),
            ])
          ),
        ]);

        // Create a `getData` function with 1 argument `data_index`
        let func = t.functionDeclaration(
          t.identifier("getData"),
          [t.identifier("data_index")],
          funcBody
        );

        // Insert the function at the beginning
        path.node.body.unshift(func);
      },

      // 2. Bypass nodes of type `MemberExpression`. Replace properties with `getData` calls
      // For example `document.createElement` will be `document[getData(0)]]
      MemberExpression({ node }) {
      // Avoid nodes that have already been “checked out” and where the `data_index` property is present, so as not to affect the new `getData` function.
        let prop = node.property.name;
        if (node.computed) return;
        if (prop == "data_index") return;

        // Put this property into the “storage” in `getData`
        data.push(t.stringLiteral(btoa(prop)));

        // Replace the property with a `getData` function call with the appropriate index
        node.property = t.callExpression(t.identifier("getData"), [
          t.numericLiteral(data.length - 1),
        ]);

        // Make the property computable
        node.computed = true;
      },


      // 3. Bypass nodes of `StringLiteral` type. Replacing strings with `getData` calls
      StringLiteral(path) {
        // Put the string into the “storage” in `getData`
        data.push(t.stringLiteral(btoa(path.node.value)));

        // Create a call to the `getData` function with the appropriate index
        const c = t.callExpression(t.identifier("getData"), [
          t.numericLiteral(data.length - 1),
        ]);

        // Replace this node with a newly created node
        path.replaceWith(c);
      },
    },
  };
}

module.exports = transformCode;

I have already described a little bit the work of this plugin in the code comments, but let's briefly describe step by step what it does:

We create an array data in which will store all properties and strings to be replaced in the code. This array will be used in the getData function that returns our data;
Next, we traverse the AST and find the root node Program, using which the getData function (returns properties and strings at a given index) will be inserted at the beginning of our code;
Then we bypass the nodes of type MemberExpression. We replace properties with calls to the getData function. In this case, constructs like document.createElement will be turned into document[getData(0)], thanks to calculated properties. Along the way, we put the names of the properties into the data array;
Finally, we bypass nodes of type StringLiteral, where we also replace strings with a call to getData with the desired index.

It is worth mentioning that parsing operations are not performed sequentially, but as the necessary node is found during AST processing.

As a result of executing this plugin, we will get the following code:

function getData(data_index) {
  let data = ["Y3JlYXRlRWxlbWVudA==", "Y2FudmFz", "Z2V0Q29udGV4dA==", "d2ViZ2w=", "Z2V0UGFyYW1ldGVy", "UkVOREVSRVI=", "Z2V0UGFyYW1ldGVy", "VkVORE9S", "bGFuZ3VhZ2Vz", "bmF2aWdhdG9y", "Z3B1", "bGFuZ3M=", "bG9n", "c3RyaW5naWZ5"];
  return atob(data[data_index]);
}

function hjj() {
  let bq = document[getData(0)](getData(1));
  let c = bq[getData(2)](getData(3));
  const a_x = c[getData(4)](c[getData(5)]);
  const nry = c[getData(6)](c[getData(7)]);
  return [a_x, nry];
}

function sztc() {
  return window[getData(9)][getData(8)];
}

let _o = {};
_o[getData(10)] = hjj();
_o[getData(11)] = sztc();
console[getData(12)](JSON[getData(13)](_o));

As you can see from the resulting code, all properties have been replaced by getData function calls with a given index. We did the same thing with strings and started to get them through function calls. The property names and strings themselves were encoded with base64 to make them more difficult to notice...

I guess you have already noticed - this plugin, and the code in general, has flaws at this stage. For example, the following things could be corrected:

The functions returning our properties and strings scream about their purpose - getData. But the good thing is that this point can be corrected by applying our first plugin, which renames identifiers.
The strings themselves inside the getData function are not reliably protected, it is quite easy to find their initial values, because it is only base64. It is more challenging to solve this problem, for example, you can remake the getData function and apply encryption instead of the well-known encoding.
The getData function is the only one, and it is not difficult to write a script that will replace all its calls with the original value by pulling and executing the function itself.

Despite all this simplicity and downsides, I think it can already be called obfuscation. But then again, how do we differ from the open-source obfuscators, since they do similar things?

We've got to remember the original problem — those obfuscations were a piece of cake for public deobfuscators. Now, let's take that code we got and deobfuscate it in webcrack! (hopefully it still can't tackle our spell🤞). I guess you could say the practical importance has been achieved - our “protected” code can no longer be pulled back in one click via a public deobfuscator

Bonus. Deobfuscation

Now let's learn a brand-new spell. Although public deobfuscators are not able to handle our plugins, however, having studied the actual concept of our obfuscation we can notice some patterns that can be used to restore the source code.

Let's get into it, and specifically take advantage of:

Having a single invoked function that stores our properties and strings;
The calls to this function are not masked;

Given these disadvantages, we can implement the following plugin:

// Name of the function, which retrieves properties and strings
let functionName = "getData"

// Function copied from the obfuscated code 
function getData_copy(data_index) {
  let data = ["Y3JlYXRlRWxlbWVudA==", "Y2FudmFz", "Z2V0Q29udGV4dA==", "d2ViZ2w=", "Z2V0UGFyYW1ldGVy", "UkVOREVSRVI=", "Z2V0UGFyYW1ldGVy", "VkVORE9S", "bGFuZ3VhZ2Vz", "bmF2aWdhdG9y", "Z3B1", "bGFuZ3M=", "bG9n", "c3RyaW5naWZ5"];
  return atob(data[data_index]);
}


function transformCode(babel) {
  const { types: t } = babel;
  return {
    name: "deobf-str-props",
    visitor: {
      // 1. remove the `getData` function from the code
      FunctionDeclaration(path){
        if(path.node.id.name !== functionName) return
        path.remove()
      },

      // 2. Go through all calls with the name `getData`
      // Call the copied function with the current argument
      // Replace the call with the obtained result
      CallExpression(path) {
        if(path.node.callee.name !== functionName) return

        let index = path.node.arguments[0].value
        let str = t.stringLiteral(getData_copy(index))
        path.replaceWith(str)        
      }
    },
  };
}

module.exports = transformCode;

Let's describe the functionality of this deobfuscation plugin:

We copied the getData function from the obfuscated code, by executing it with the required argument (index) we can get the required string;
We went through all the getData function calls and replaced them with the result of its execution;
Finally, we found the getData function in AST and removed it from the code, since it is no longer needed there.

As a result, we get the following code:

function hjj() {
  let bq = document["createElement"]("canvas");
  let c = bq["getContext"]("webgl");
  const a_x = c["getParameter"](c["RENDERER"]);
  const nry = c["getParameter"](c["VENDOR"]);
  return [a_x, nry];
}

function sztc() {
  return window["navigator"]["languages"];
}

let _o = {};
_o["gpu"] = hjj();
_o["langs"] = sztc();
console["log"](JSON["stringify"](_o));

Thus, we were able to get rid of obfuscation that hides properties and strings by writing a simple plugin for babel using the shown disadvantages.

I hope this small example explained how you can fight such nuisances with the help of babel. Using these approaches, you can also solve more complex obfuscations - the main thing is to find patterns in the code and skillfully operate with AST.

Conclusion

We've learned about obfuscation, a technique that complicates code reverse engineering, and the tools to implement it. Although there are public solutions that obfuscate JavaScript code, there are just as many public solutions that can remove this protection in an instant.

Therefore, you need to write your own solutions to protect code that can't be removed by public deobfuscators. One reliable way to implement obfuscation in JS is to write custom babel plugins that interact with the AST of the desired code, turning it into a less readable form.

Of course, this area has known techniques and approaches to obfuscation, but nevertheless remains open to creativity and new “tricks” that can potentially make learning the code more difficult. Despite the large number of such techniques, they do not guarantee the secrecy of algorithms at all, because code is always “in the hands” of the client. Besides, there is a possibility of debugging, which can make it easier to study the code. Obfuscation allows you to rather turn away poorly motivated researchers, thus increasing the cost of reverse engineering.

There are some advanced approaches, for example, one of them among obfuscation is virtualization of code, or simply speaking, creating a virtual machine in JS that will execute custom bytecode. This approach almost completely removes the chance of static analysis and makes debugging as difficult as possible. However, this is a separate subject for discussion 🍺....

I hope it was useful for you to get information on this topic, and you won't blame yourself or your programmers for initially obfuscated code anymore. Appreciate these wizards 🧙🏻‍♀️! I will be glad to discuss with you the latest trends in magic here😏

Best regexp alternative for Go. Benchmarks. Plots.

Rustem Kamalov — Sat, 26 Aug 2023 05:10:24 +0000

Introduction

"Don't use regular expressions, otherwise you'll have 2 problems instead of 1" - that's what the experts say. And what is left for the naughty ones who want to efficiently search through a huge number of templates?

Sure, there are cool solutions like Ragel or re2c for this specific problem. However, for my project it seemed impractical to master these fine technologies for the time being.

In this article, we will look at alternatives to the standard regexp library in Go and benchmark them for speed and memory consumption. We will also consider the differences between them from a practical point of view.

Analogues

At the moment, I've found the following working alternatives to the default regexp that can be used to find patterns in Go (the versions used in the benchmarks are given in brackets):

go-re2 (1.3.0) - as simple as possible replaces the default regexp. Uses C++ re2 to improve performance when dealing with large inputs or complex expressions;
regexp2 (1.10.0) - a feature-rich regexp engine for Go. It does not have runtime guarantees like the built-in regexp package, but is compatible with Perl5 and .NET;
go-pcre (1.0.0) - provides support for Perl compatible regular expressions using libpcre or libpcre++. JIT compilation is available, making this fork much faster than its counterparts. On the downside, you'll need the libpcre3-dev dependency;
rure-go (regex 1.9.3) - uses the Rust regex engine with CGo bindings. The downside is a Rust library dependency that needs to be compiled;
gohs (1.2.2 + hs5.4.1) - regex engine designed for high performance. It is implemented as a library that provides a simple C-API. It also requires compilation and linking of third-party dependencies;
go-yara - A tool for identifying and classifying malware samples. Although YARA has functionality for templating and regular expressions, it is very limited, so I will not include this library in the upcoming tests.

Existing benchmarks

Before we start comparing the aforementioned solutions, it is worth to show how bad things are with the standard regex library in Go. I found the project where the author compares the performance of standard regex engines of various languages. The point of this benchmark is to repeatedly run 3 regular expressions over a predefined text. Go came in 3rd place in this benchmark! From the end....

Language	Email(ms)	URI(ms)	IP(ms)	Total(ms)
Nim Regex	1.32	26.92	7.84	36.09
Nim	22.70	21.49	6.75	50.94
Rust	26.66	25.70	5.28	57.63
...	...	...	...	...
Python PyPy3	258.78	221.89	257.35	738.03
Python 3	273.86	190.79	319.13	783.78
Go	248.14	241.28	360.90	850.32
C++ STL	433.09	344.74	245.66	1023.49
C# Mono	2859.05	2431.87	145.82	5436.75

As far as I can tell, benchmarking regex engines is not the easiest topic, because you need to know implementation and algorithm details for a correct comparison. From other benchmarks I can highlight the following:

Benchmarks Game - comparison of engines with optimized versions for each language. For example, Go is no longer at the bottom, but it is still far from ideal there... And of course it does not use a native library, but a wrapper over PCRE - go-pcre, which was specified in analogues.
Performance comparison of regular expression engines - Comparison of different regex engines (PCRE, PCRE-DFA, TRE, Oniguruma, RE2, PCRE-JIT).
A comparison of regex engines - here the author tries to compare engines using different regular expressions, which can complicate things depending on the engine implementation. In this benchmark, the author's top three engines are: Hyperscan, PCRE (with JIT compilation) and Rust regex (rure uses it):

Benchmark #1

Now let's try to compare the analogues with default regex engine libraries of other languages. And also see how much faster they are compared to the default Go regex. To do this, I updated the above project by adding new libraries code. Here is what I got after running it on my machine:

Language	Email(ms)	URI(ms)	IP(ms)	Total(ms)
Go Rure	2.61	2.11	3.33	8.05
C++ SRELL	1.74	3.03	10.67	15.45
Rust	11.66	1.70	5.13	18.48
Nim	13.98	14.35	3.13	31.46
PHP	14.43	14.63	4.87	33.93
Go PCRE	14.18	14.98	6.21	35.37
C# .Net Core	10.83	5.10	26.71	42.64
Javascript	42.78	30.17	0.92	73.87
Go Re2	35.81	37.86	33.79	107.46
Go Hyperscan	90.17	31.64	8.68	130.49
Perl	92.51	66.42	22.51	181.44
Dart	73.48	63.60	80.52	217.59
C PCRE2	110.87	100.57	10.50	221.94
Kotlin	120.31	163.53	293.69	577.53
Java	205.14	201.55	295.68	702.36
Python 3	246.26	176.01	303.08	725.34
C++ STL	328.87	273.43	230.35	832.64
Go	270.19	275.73	504.79	1050.71
Go Regexp2	1703.51	1482.60	64.46	3250.57
C# Mono	2543.82	2139.44	110.37	4793.64

Ps. Shortened the table a bit, so please check the project fork to see all the results..
Pss. In this case, virtualization might have affected the results (the tests were performed in a Docker environment on Windows 11).

Still, you can see how much faster regexps can be with some libraries! We even outperformed Rust with a Go library that uses the Rust library 🥴🙆🏼‍♂️. Perhaps this is what the author of this solution is trying to explain to us in his repository.

As a result, almost all alternative solutions give us a speedup of 8-130 times! Except for Regexp2, which turns out to be slower than the standard library.

Benchmark #2

Questions

While studying the existing benchmarks and the results of Benchmark#1, I was lacking the answers to the following questions:

How fast do the above libraries process large files?
How much faster is the processing when grouping regular expressions?
How fast will regular expressions with no matches in the text be processed?
How much memory do the different libraries use?
How many regexps will I be able to compile using grouping?

Benchmarker differences

To answer these questions, I wrote a small benchmarking program that can be used to compare different regex engines in terms of speed and memory usage. If you want to test it yourself or evaluate the correctness of the used methods, here is the code.

The following features of this benchmark are worth mentioning:

In the tests below, I used 5 different regular expressions:

allRegexps["email"] = `(?P<name>[-\w\d\.]+?)(?:\s+at\s+|\s*@\s*|\s*(?:[\[\]@]){3}\s*)(?P<host>[-\w\d\.]*?)\s*(?:dot|\.|(?:[\[\]dot\.]){3,5})\s*(?P<domain>\w+)`
allRegexps["bitcoin"] = `\b([13][a-km-zA-HJ-NP-Z1-9]{25,34}|bc1[ac-hj-np-zAC-HJ-NP-Z02-9]{11,71})`
allRegexps["ssn"] = `\d{3}-\d{2}-\d{4}`
allRegexps["uri"] = `[\w]+://[^/\s?#]+[^\s?#]+(?:\?[^\s#]*)?(?:#[^\s]*)?`
allRegexps["tel"] = `\+\d{1,4}?[-.\s]?\(?\d{1,3}?\)?[-.\s]?\d{1,4}[-.\s]?\d{1,4}[-.\s]?\d{1,9}`

In a good way, I should have used tricky regular expressions, like other benchmark authors do - to check the "weaknesses" of the algorithms. But I don't have much insight into the under-the-hood details of engines, so I used general-purpose regexps. That's why I think it should be possible to evaluate different parameters of libraries from a practical point of view.

Instead of a static file, we will use a string containing our matches, repeated many times in memory to simulate files of different sizes:

var data = bytes.Repeat([]byte("123@mail.co nümbr=+71112223334 SSN:123-45-6789 http://1.1.1.1 3FZbgi29cpjq2GjdwV8eyHuJJnkLtktZc5 Й"), config.repeatScanTimes)

In addition to running the regexps sequentially over the data, there will be a separate run with named grouping of regular expressions, where they will take the following form:

`(?P<name_1>regexp_1)|(?P<name_2>regexp_2)|...|(?P<name_N>regexp_N)`

By the way, Hyperscan has a special functionality where we can build a database of regexps and use it against data. In the benchmarks I will use this method.

Unlike in Benchmark#1, for each regexp I will measure the time to find the result, without taking into account the compilation time;
In the end, we will get the results for each library and each regexp in the following form:

Generate data...
Test data size: 100.00MB

Run RURE:
  [bitcoin] count=1000000, mem=16007.26KB, time=2.11075s 
  [ssn] count=1000000, mem=16007.26KB, time=62.074ms 
  [uri] count=1000000, mem=16007.26KB, time=69.186ms 
  [tel] count=1000000, mem=16007.26KB, time=83.101ms 
  [email] count=1000000, mem=16007.26KB, time=172.915ms 
Total. Counted: 5000000, Memory: 80.04MB, Duration: 2.498027s
...

1-2. Processing large files and grouping regular expressions

By "large file" I mean the amount of data generated from our predefined string in memory equal to 100MB. Of course it's hardly a big file, but I didn't want to wait hours for results in some cases 😅.

Also, it makes sense to try grouping all regular expressions into one and see how much it affects performance. The hypothesis is that a sequential run of regexps over the data should take longer than a single run with a grouped regexp.

Well, let's run the benchmark and see how fast each library processes the data. You can do this with my tool as follows:

# Bench all libs, repeat the line 1000000 times (100MB)
regexcmp 1000000 
# Bench individual lib
regexcmp 1000000 rure
# See all options
regexcmp -h

As a result, we have the following data:

Lib	Email	Tel	URI	SSN	Bitcoin	Total	Total-group
Rure	0.173s	0.083s	0.069s	0.062s	2.11s	2.49s	10.13s
PCRE	49.5s	0.367s	2.92s	2.07s	2.19s	57.07s	9.94s
Re2	0.954s	0.63s	0.956s	0.945s	1.05s	4.54s	3.24s
Hyperscan	0.469s	1.09s	0.669s	0.174s	1.97s	4.38s	4.97s
Regexp2	126.4s	1.02s	21.51s	2.63s	3.38s	154.9s	20.65s
Go regexp	11.22s	1.52s	3.62s	2.66s	3.32s	22.36s	26.49s

The plot below shows the processing time of 100MB of data by all regexps in sequential mode and using grouping:

Conclusions:

Grouping can indeed significantly improve execution speed, but in some cases it can make it worse :);
The fastest in sequential processing turned out to be - Rure, with grouping - Re2;
Certain regexps may cause problems with some libraries (time to find email in Regexp2 and PCRE);
Now it is hard to say that some solutions are 180 times faster than the standard library, the maximum gain is x8-9.

3. Non-matching regexps

In the previous case, we simulated the ideal situation where there are always matches in the data. But what if there are no hits for regexp in the text, how much will that affect performance?

In this test, I additionally added 5 modified regexps for SSN that do not match the data. In this case, SSN means \d{3}-\d{2}-\d{4} regexp, and Non-matching - \d{3}-\d{2}-\d{4}1. Not a big difference? But let's see how it affects the time it takes to find all the matches:

Lib	SSN	Non-matching	Total	Total-group
Rure	0.06s	0.083s	2.93s	15.68s
PCRE	2.06s	4.21s	81.02s	13.52s
Re2	0.960s	0.449s	6.75s	3.26s
Hyperscan	0.175s	0.043s	4.61s	4.94s
Regexp2	2.73s	3.09s	171.1s	15.72s
Go regexp	2.66s	2.57s	35.13s	37.66s

The plot below shows the time taken to process all 10 regexps, sorted by Non-matching processing time:

Conclusions:

This time is the same: the fastest in sequential processing was - Rure, with grouped expressions - Re2;
PCRE is once again different, with 2x time to process non-matching regexps in sequential mode;
Some algorithms are much faster when there are no matches (Re2, Hyperscan);

4. Memory consumption

Now let's see how much memory the different solutions consume when processing a 100MB file. Below, I have provided the results for each individual regexp and the total amount of memory consumed:

Lib	Email, MB	Tel, MB	URI, MB	SSN, MB	Bitcoin, MB	Non-matching, KB	Total, GB	Total-group, GB
Rure	16	16	16	16	16	0.0001	0.08	0.08
PCRE	186	186	186	186	186	0.38	0.93	0.9
Re2	254	254	254	255	254	100473	1.77	0.97
Hyperscan	314	314	314	314	314	0.68	1.57	1.7
Regexp2	997	854	854	854	902	396006	6.44	5.23
Go regex	191	144	144	160	208	3.62	0.78	1.8

The graph below shows the memory used by the libraries to process 10 regexps (as in the last test), sorted by the time of `non-mathcing':

Conclusions:

Rure surprises with its practically zero memory consumption;
Regexp2 is resource hungry, consuming significantly more memory than its competitors;
Re2, despite its speed, is not the most memory-efficient solution;
Go regex is on the good side, with relatively low costs in sequential mode.

5. Maximum number of regexps

The main questions seem to be answered. Now let's see the maximum number of regexps we can compile with different solutions. In this case we will take a single regexp and repeat it many times in groups.
For this purpose I will use the URI regexp:

go[\w]+://[^/\s?#]+[^\s?#]+(?:\?[^\s#])?(?:#[^\s])?

Next, I've listed the results of compiling the regexps, along with the memory they use. The numbers in the first line are the number of URI expressions in the group:

Lib	100	250	500	1000	5000	10000
Rure	0.38MB	❌	❌	❌	❌	❌
PCRE	0.40MB	2.37MB	❌	❌	❌	❌
Re2	7.10MB	15.51MB	38.35MB	92.79MB	1181MB	❌
Hyperscan	0.03MB	0.06MB	0.12MB	0.25MB	1.21MB	2.52MB
Regexp2	1.06MB	3.96MB	12.56MB	43.93MB	926MB	3604MB
Go regex	1.29MB	4.52MB	14.02MB	47.18MB	942MB	3636MB

Conclusions:

As we can see, some solutions have limitations on the size of compiled regexps;
Hyperscan not only allows to use a large number of regexps, but also uses minimal memory for compiled regexps;
Regexp2 and Go Regex have comparable memory consumption and also allow to compile a large number of regexps;
Re2 consumes the most memory at compile time.

Conclusion

I hope it was useful for you to learn about alternative solutions for regexps in Go, and based on the data I presented, everyone will be able to draw some conclusions for themselves, which will allow you to choose the most suitable regex solution for your situation.

We can compare these libraries, the algorithms they use, their best/worst sides in details and for a long time. I just wanted to show the difference between them in the most general cases.

Thus, I would advise you to look at rure-go for maximum speedup of regexps, but if you need the easiest library installation without dependencies, that is go-re2. And in the case of handling a large number of regexps hyperscan would be a good choice. Also, don't forget the cost of using CGo in some libraries.

As for me, I will definitely use these "findings" in my future project :).

Googling for free 🎉: open source solution with API

Rustem Kamalov — Thu, 29 Jun 2023 18:46:46 +0000

Init

For free? Of course, no one is going to take money from you for "Googling"... But what if you want to automate the process? That's where the problems lie, and where the business opportunities are.

Like in the last article, I went to the Internet, this time to get the files from Google. The story repeated itself and I realized that I had the code for this somewhere in the stash. I was happy to start studying it and realized that it wasn't much fun and it was easier to find something already done...

Next, we'll take a quick look at what solutions the market has to offer at the moment, and what kind of bugs are being given away for free. We'll also look at the problems of automating this process and how it can be done easily and for free!

Comparing Solutions and Problems

Problem 0. No problem

There is a whole industry dedicated to providing SERP (search engine results page). For example, Google is ready to provide API to search itself, while other services provide search results from Yandex, Baidu and other services in addition to access to Google. Here is a comparison of some commercial solutions:

Google Custom Search - 100 free queries per day. $5 for 1000 queries;
SerpApi - 100 free queries per month. Plans range from $50 to $250 per month;
SerpWow - 100 free queries. Plans from $25 to $9000 per month;
Oxylabs - 1 free week with 5k queries. Plans range from $49 to $249 per month.

There are about a dozen similar services. The market situation is generally clear: there are trial periods, if you like it, study the documentation, stick with the service, buy and renew. The prices are not sky-high, but what about open source projects? 🤓

Problem 1. Solutions are outdated

First and foremost, I would like to find a solution written in Go to use in my code, or adapt it if possible. Let's take a look at what's available on the ~~flea market~~ GitHub:

github.com/rocketlaunchr/google-search - Yay! Immediately found a google solution for Go. But to my great regret, it refused to work and returned no results ☹️. The developers warn us about this (fixed as of this writing): The search engine DOM is subject to change, so solutions of this kind need to be constantly tweaked.
github.com/jarun/googler - A once popular console search toolkit written in Python. Last commit in 2021, the repository is archived, and it looks like the solution no longer works.
github.com/ecoron/SerpScrap - Library for Python. Failed to deploy it - it is necessary to install additional dependencies, how to use its docker is not documented. The solution is not developed since 2019 - do not have high hopes, move on...
github.com/serphacker/serposcope - Obsolete utility in Java with some UI for SEO. Seems to have Captcha solving functionality 🧐. Doesn't provide an API.
github.com/Athlon1600/SerpScraper - Solution for PHP. Not sure if it works (last commit was 3 years ago), but there is some interesting code and captcha solving functionality.

Problem 2: Fast detection without a browser

Some of the solutions I've described and seen use HTTP clients that are easy to detect. If we use them, after a short time we will encounter captcha and our program will be bullied and called a bot 😢. The best implementations use a browser to search and then parse the page, but even this approach is not a panacea.

By the way, you can use the following services to check if you are a bot and benchmark your solutions:

bot.sannysoft.com - shows the general parameters of your client and performs some tests;
whatismybrowser.com - shows if the reported browser is the real one;
abrahamjuliot.github.io/creepjs - able to detect hiding methods.

Try it, maybe as a certified androiud you should already have an electric sheep... 🐑⚡️.

Problem 3. They are just interfaces/wrappers :)

Some of the libraries I found turned out to be just interfaces to paid services. For example:

github.com/towfiqi/serpbear - Something for SMM people. It's just a wrapper that uses the API of popular services.
github.com/serpapi/google-search-results-python - Python library to interact with a paid service.

Uh-oh, what to do? There has to be a free and bad counterpart to every commercial service, right? Wrong.

Let's solve problems

Separate the wheat from the chaff

Let's automate the SERPs ourselves. On paper, solving problems does not look complicated. For example, to find all search results on Baidu page, we can use this simple CSS selector (you can run it in the developer console in your browser)

// Find all elements of `div` that contain classes `c-container` and `new-pmd`
document.querySelectorAll(`div.c-container.new-pmd`)

You can do the same with XPath:

$x(`//div[contains(@class, "c-container") and contains(@class, "new-pmd")]`)

Then all that remains is to extract the useful information from the resulting HTML blocks and present it in a usable form.

Hide and automate

How do I automatically launch and parse CSS selectors? For many programming languages there are frameworks that interact with the browser, such as the popular Selenium or Puppeter. In my case, I paid attention to Go-Rod.

As we have already found out, in terms of hiding our automated activity, it is better to do it via the browser than simply using an HTTP client. By the way, Go-Rod has an extension stealth that will additionally help hide our activity.

So, all the pieces are in place. All that remains is to design the right search query. You can learn this on your own - by experience or by looking at tutorials, e.g. here is a description of parameters for Baidu. Next, we use Go-Rod to start the browser in headless mode, navigate to the constructed URL and parse the results with CSS selectors... Voilà!

The flip side of the coin

Of course, even using the browser and tricks to hide our activity - sooner or later we will bite the Captcha. For the criterion of usefulness of the solution, I invented benchmark 101 - if you can make 101 free requests per day (Google provides 100) through one machine (1 IP), then the solution ~~surpasses all analogues~~ is useful. And considering that there are 3 automated services, it's triple useful!

The disadvantages of this approach are:

We rely on the persistence of search engines DOM pages - if their structure changes, we have to fix our solution. But it depends more on the page elements you choose to "hook on" - for instance, the code I wrote 4 years ago was still able to parse results from a Google page unchanged (but was quickly detected as a bot).
By using the browser instead of a regular HTTP client, we have a slightly higher resource requirement and additional dependencies are needed.

Obviously, the way to work with different search search engines will be different. The solution I will describe below uses a unified approach, so let's see how to get the search API in 2 clicks....

API Search Results

I used the above approach in the OpenSERP project. In order to enable access to search results via API, you can:

Run OpenSERP in a Docker environment:

docker run -p 127.0.0.1:7000:7000 -it karust/openserp serve -a 0.0.0.0 -p 7000

Or use compiled version for your OS, but you will need to have Chrome on your machine.

Running our server, we can use the following query to get 25 search results in German for the query how to get banned from google fast:

http://127.0.0.1:7000/google/search?text=how to get banned from google fast&lang=DE&limit=25

An example of the result in JSON format 🤗:

[
    {
        "rank": 1,
        "url": "https://www.apprimus-verlag.de/catalogsearch/result/index/?p=5&q=aiab.top%5BTutorial+on+how+to+get+Google+to+index+your+site+fast",
        "title": "aiab.top[Tutorial on how to get Google to index your site fast",
        "description": "Artikel 61 - 75 von 288 — Suchergebnisse für: \"aiab.top[Tutorial on how to get Google to index your site fast\" · Development of an Algorithm for the Taktline Layout ..."
    },
    {
        "rank": 2,
        "url": "https://chrome.google.com/webstore/detail/ban-checker-for-steam/canbadmphamemnmdfngmcabnjmjgaiki?hl=de",
        "title": "Ban Checker for Steam - Chrome Web Store",
        "description": "Automatically check bans of people you recently played with, your friends, and group members. ... Go to extension's options page to do so."
    },
    {
        "rank": 3,
        "url": "https://www.reuters.com/article/usa-google-idDEKBN28O1T1",
        "title": "Einige Google-Dienste vorübergehend nicht erreichbar",
        "description": "14.12.2020 — Viele Google-Nutzer schauten am Montag in die Röhre. Mehrere Dienste des US-Technologiekonzerns Alphabet wie die Videoplattform YouTube, ..."
    },
...
]

In order to use Yandex or Baidu, we use the appropriate endpoint - yandex or baidu instead of google in the query. The parameters of the queries and their description are presented on the repository page.

It is also possible to get the results via the CLI.

Conclusion

Despite the availability of paid options to automate searches through well-known engines - there should always be a free alternative. That's why I created an open-source solution OpenSERP that supports multiple search engines (Google, Yandex, Baidu), can be deployed on your hardware and helps you avoid being bound to a paid 3rd-party service.
Of course, it still needs some work before the solution can be used on an industrial scale. But in any case I hope it can be useful for the reader, so I invite you to check out the project :)

PS

You may also be interested in reading about automating web archives searches.

Get CIA certificate by exploring archives. Golang solution for Web Archive data extraction.

Rustem Kamalov — Thu, 01 Jun 2023 02:38:22 +0000

You may be wondering, "How do I retrieve data from a resource that has long been closed or some of the data from it has been deleted?" I had the same question a long time ago, that's when I fully discovered the capabilities of Web Archives and created a web archive extraction tool based on Common Crawl.

Recently I had to solve a similar problem, but Common Crawl seemed to be barely breathing due to overload... So I decided to reinvent the wheel and improve my tool by extending it with a Wayback Machine archives source. While developing it, I accidentally got certified by the CIA. Now my altruistic self wants to show everyone how to get such a cool thing on their virtual shelf!

For those who are interested in the "behind the scenes" details, I also described what the two aforementioned services have in common and how we can use them through the API. At the end, we will get the certificate using the GoGetCraw tool.

Web Archive services

First, it is worth mentioning the general points of services such as Common Crawl and Wayback Machine... Their archives are stored in WARC/ARC files, also called CDX files (probably from Capture/Crawl inDeX). Then, with a tool like pywb, we can serve these indexes and examine them with various filters.

For example, in the screenshot below, you can see the result of the request http://<server_addr>/cdx?url=https://twitter.com/internetarchive/ to the CDX server:

As you can see, the result will be a number of rows related to the queried URL (https://twitter.com/internetarchive/). This data contains the status at the time of the request (status code) and the data collection time (timestamp), the file type (mimetype) and other interesting parameters. A more detailed description of the CDX server and the parameters used in the queries can be found here.

In addition to the Wayback Machine and Common Crawl services described below, there are many others. Unfortunately, their archives are less extensive and are usually archives of individual country websites or dedicated to a topic (e.g. art). You can find some of them here.

Let's exctract some files

I believe some of you have already used web archives through pretty interfaces. Now I will show you how to do it a fun way, using the API of Wayback Machine. For example, we want to get all the JPEG files on cia.gov and its subdomains and then download the file we are interested in.

To perform our task with this resource, we construct the following query:

https://web.archive.org/cdx/search/cdx?url=*.cia.gov/*&output=json&limit=10&filter=mimetype:image/jpeg&collapse=urlkey

Where:

/cdx/search/cdx - the endpoint of the CDX server,
`url=.cia.gov/` - our target domain,
filter=mimetype:image/jpeg - filtering by MIME type of JPEG file,
output=json * - demand result in JSON format, limit=10 - limit to 10 results, collapse=urlkey - get unique URLs (without this, there are many duplicates).

As a result, we get the 10 images found in the archive. In addition to the URLs, the response contains the MIME type of the files (useful if you do not use filtering), as well as the status code when accessing the object at the time of archive creation:

[["urlkey","timestamp","original","mimetype","statuscode","digest","length"],
["gov,cia)/++theme++contextual.agencytheme/images/aerial-analysis-btn.jpg", "20150324125120", "https://www.cia.gov/++theme++contextual.agencytheme/images/aerial-analysis-btn.jpg", "image/jpeg", "200", "OJRFXPWOPZQGPRIZZQZOTRSZKAVLQLKZ", "3845"],
["gov,cia)/++theme++contextual.agencytheme/images/aerial_cropped.jpg", "20160804222651", "https://www.cia.gov/++theme++contextual.agencytheme/images/Aerial_Cropped.jpg", "image/jpeg", "200", "3WII7DZKLXM4KSQ5UTEKO5EL7H5VTB35", "196685"],
["gov,cia)/++theme++contextual.agencytheme/images/background-launch.jpg", "20140121032437", "https://www.cia.gov/++theme++contextual.agencytheme/images/background-launch.jpg", "image/jpeg", "200", "3C4G73473VYPOWDNA4VJUV4Q7EC3IXN4", "44501"],
["gov,cia)/++theme++contextual.agencytheme/images/background-video-panel.jpg", "20150629034524", "https://www.cia.gov/++theme++contextual.agencytheme/images/background-video-panel.jpg", "image/jpeg", "200", "CQCUYUN5VTVJVN4LGKUZ3BHWSIXPSCKC", "71813"],
["gov,cia)/++theme++contextual.agencytheme/images/bannerheads/an-1.jpg", "20130801151047", "https://www.cia.gov/++theme++contextual.agencytheme/images/bannerheads/an-1.jpg", "image/jpeg", "200", "GPSEAEE23C53TRGHLMBXHWQYNB3EGBCZ", "14858"],
["gov,cia)/++theme++contextual.agencytheme/images/bannerheads/an-2.jpg", "20130801150245", "https://www.cia.gov/++theme++contextual.agencytheme/images/bannerheads/an-2.jpg", "image/jpeg", "200", "L6P2MNAAMZUMHUEHJFGXWEUQCHHMK2HP", "15136"],
["gov,cia)/++theme++contextual.agencytheme/images/bannerheads/an-3.jpg", "20130801151656", "https://www.cia.gov/++theme++contextual.agencytheme/images/bannerheads/an-3.jpg", "image/jpeg", "200", "ODNXI3HZETXVVSEJ5I2KTI7KXKNT5WSV", "19717"],
["gov,cia)/++theme++contextual.agencytheme/images/bannerheads/an-4.jpg", "20130801150219", "https://www.cia.gov/++theme++contextual.agencytheme/images/bannerheads/an-4.jpg", "image/jpeg", "200", "X7N2EIYUDAYWMX7464LNTHBVMTEMZUVN", "20757"],
["gov,cia)/++theme++contextual.agencytheme/images/bannerheads/banner-benefits-background.jpg", "20150510022313", "https://www.cia.gov/++theme++contextual.agencytheme/images/bannerheads/banner-benefits-background.jpg", "image/jpeg", "200", "VZJE5XSAQWBD6QF6742BH2N3HOTSCZ4A", "12534"],
["gov,cia)/++theme++contextual.agencytheme/images/bannerheads/chi-diversity.jpg", "20130801150532", "https://www.cia.gov/++theme++contextual.agencytheme/images/bannerheads/CHI-diversity.jpg", "image/jpeg", "200", "WJQOQPYJTPL2Y2KZBVJ44MVDMI7TZ7VL", "6458"]]

Next, to access the archived file, we take one of the results above and use the following query:

https://web.archive.org/web/20150324125120id_/https://www.cia.gov/++theme++contextual.agencytheme/images/aerial-analysis-btn.jpg

Where:

/web - file-server endpoint,
20230502061729id_* - timestamp obtained during the previous query + id_,
https://www.cia.gov/++theme++contextual.agencytheme/images/aerial-analysis-btn.jpg* - file URL received during the previous request.

GoGetCrawl the CIA certificate

It is worth saying that at the moment there are already a few tools for interacting with archives. But with a cursory glance I haven't found any that, apart from URL mining, could download the file and easily integrate into other Go projects!
With this in mind, I updated my outdated solution by doing some refactoring and adding Wayback Machine as a second source of archive data.

Let's get the paper

Finally, we will get a certificate using gogetcrawl. You can use it in several ways described here. You won't have to compile or install anything, and so for convenience:

You can download the latest release. And use the binary as follows:

gogetcrawl file *.cia.gov/* --dir ./ --ext pdf

After waiting a little bit we should get the cherished file, if you are bored or there is no file, you can see what happens by adding the -v flag.

You can as well use Docker:

docker run uranusq/gogetcrawl url *.cia.gov/* --ext pdf

As a result of this command, we should see the URL with the PDF file. You can learn more about the commands and possible arguments by using the -h flag.

Installation option for Go-phers:

go install github.com/karust/gogetcrawl@latest

Congrats

We received our certificate 🫢. For those who don't have it yet, it looks like this:

Not fully understanding what it is and why it is there, I immediately rushed to share this opportunity with you.

Open-source

For those who use Go and want to become an archivarius, there is an option to apply GoGetCrawl in your project:

go get github.com/karust/gogetcrawl

For example, a minimal program that will show us all the pages of example.com and its subdomains with the status of 200 looks like this:

package main

import (
    "fmt"

    "github.com/karust/gogetcrawl/common"
    "github.com/karust/gogetcrawl/wayback"
)

func main() {
    // Get only 10 status:200 pages
    config := common.RequestConfig{
        URL:     "*.example.com/*",
        Filters: []string{"statuscode:200"},
        Limit:   10,
    }

    // Set request timout and retries
    wb, _ := wayback.New(15, 2)

    // Use config to obtain all CDX server responses
    results, _ := wb.GetPages(config)

    for _, r := range results {
        fmt.Println(r.Urlkey, r.Original, r.MimeType)
    }
}

On the project page, you can find more examples, including file extraction and CommonCrawl usage.

PS

I hope no one was click baited, and everyone enjoyed the new certificate in their collection :) Perhaps there are other web archive services other than those described in the article that can be used similarly? Let me know.