Replaying network requests on puppeteer

sametcodes — Mon, 28 Nov 2022 10:04:26 +0000

You possibly tried to read the request header or response body of incoming and outgoing requests by network intercepting if you’ve made some developments on puppeteer headless browser scripts. I’ve been working on puppeteer scripts for reverse engineering and scraping purposes and realized that there is no request replaying method. I needed that ― so I made a plugin for that: https://www.npmjs.com/package/puppeteer-extra-plugin-replayer

How to replay requests?

First of all, I want to explain how it works. CDP and Devtools allow you to copy a request with its body and headers as a prepared code by using the “Copy as Fetch code” option. If you want to replay the request, it’s enough to paste it into the JavaScript console and modify the requested content if it’s needed and send it.

Voilà, you replayed a request. It’s that easy.

But, what about CORS policies? No worries about that, since you sent that code through the console ― so the requests fit into the CORS policies. If you send the request through your bash CLI as a curl command or on another tab console, your request could be rejected ― if we assume that there are proper CORS policies on the server.

The methods

There are two methods:

.catchRequest in the page object
.replay method that in the return value of .catchRequest

You can check the interface declarations to see the exact values.

.catchRequest takes two arguments. The first is a regex pattern to catch requests by their URLs, and the second is a callback method to trigger the wanted request. If it’s being triggered initially as the page opens, then page.goto() must be called in the trigger callback function.

(async () => {
  const request = await page.catchRequest({ pattern: /task.json\?taskname=login/},
        () => page.goto("https://twitter.com"));
  const response = await request.replay();
})();

If the request is being triggered as clicks on a button, then .click() function must be defined in the callback as here:

(async () => {
        await page.goto("https://twitter.com");

    const request = await page.catchRequest({pattern: /task.json\?taskname=login/},
            async () => {
                await page.waitForSelector("a[href='/login']")
        await page.click("a[href='/login']")
            });

    const response = await request.replay();
})();

.replay() can take url, method, headers, and body as optional parameters, and you can modify the request by redefining them when replaying.

const response = await request.replay({
    url: "https://twitter.com/logout", // defining a new URL is possible
    method: "POST", // changing the request method is possible as well
    body: JSON.stringify({test: true}),
    headers: {test: true},
});

If you want to get and use the default values of the request instead of overwriting, use a callback function and return the new value.

const response = await request.replay({
    url: url => url.replace(/login/, "/logout"),
    method: "POST", // changing the request method is possible as well
    body: body => body + ";test=true",
    headers: headers => {...headers, test: true}
});

One last note

You can check out the CDP Network tab if you want to see replayed requests during the development process. Notice that the initiator value of the requests is different.

Conclusion

The package helps to replay any XHR requests, but it' possible to catch and replay any requests.

If you want to see the source code and contribute to the project, please check the repository.

Happy coding!

What is JSON with padding?

sametcodes — Sun, 27 Nov 2022 21:11:50 +0000

JSONP (JSON with padding) is a tricky way to send cross-site GET requests by getting around the browser’s same-origin policy. The modern browsers follow the CORS rules when sending XHR (XMLHttpRequest) requests, but JSONP bypasses it due to the way how it sends requests.

JSONP isn’t an XHR request; it’s a script load on the page but returns a function call. This is what JSONP data looks like:

callback({
    data: {
        id: 1
    }
})

But, what does it do exactly?

Let’s assume your web page has functions from the app.js file that loads on the page with script tags. As the page opens, the script tag will send a request for the app.js and loads it, the definition of functions will be processed onto the page, and the functions will be ready to call. This is how JavaScript files are processed.

But, if you put an endpoint URL into the script tag with a callback parameter, the server would return only a JSONP object ready-to-call on the client if the returning function is defined on the client right after the page resolves the script tag.

Dynamic JSONP requests

Instead of sending the requests when the page initiates, requests can be sent dynamically by injecting a script element into the DOM within a function.

function requestServerCall(url) {
  var head = document.head;
  var script = document.createElement("script");

  script.setAttribute("src", url);
  head.appendChild(script);
  head.removeChild(script);
}

function myCallback(data){
    // process your data here
}

requestServerCall("URL?callback=myCallback")

Is it safe?

Well, no. It’s just a basic hack to make cross-domain requests, and sites that use JSONP services are prone to CSRF; well, that’s why XHR request methods have CORS policies: to keep safe client users.

An attack that prepared for potential XSS vulnerabilities, such as stealing user cookies, can also be demonstrated for JSONP. The provider of JSONP data must be trustable and ensure that it doesn’t return any malicious JavaScript code and try to run a different callback than given.

Conclusion

JSONP is an old-fashioned and hacky way, but not safe, to send GET requests to the client. XHR requests, such as ajax and fetch, are restricted to sending cross-site requests, but it is still possible to send with JSONP.

Plus, it is not that useful to use it in JavaScript applications due to no callback functions are provided to handle potential errors, even though some libraries like jQuery and Angular implemented a native way to send JSONP requests.

So, there is no need to use it.

This blog post was published on here.

DEV Community: sametcodes

Replaying network requests on puppeteer

How to replay requests?

The methods

One last note

Conclusion

What is JSON with padding?

Dynamic JSONP requests

Is it safe?

Conclusion