DEV Community: Jackson Carter

Understanding Differences Between Web Debugging Proxies

Jackson Carter — Thu, 27 Aug 2020 19:49:27 +0000

Web debugging proxies like Fiddler are a little…shall we say…complicated. Even people with computer science backgrounds don’t necessarily know how they work. This sort of obscurity can create problems for people if they’re, say, tasked with determining which piece of software would suit their or their company’s needs best.

I’ll give you a quick rundown of the three most popular options in the field: utilitarian and focused Charles, bouncy up-and-comer Proxyman, and (of course) longtime favorite Fiddler, specifically the recently-released updated version Fiddler Everywhere. The goal here is to give an understanding of the points of comparison between different proxies, so you can draw your own conclusions about which one fits your needs.

If you’re wondering what exactly a web debugging proxy is, I explain that here.

Charles

Ask anyone who works in software if they’ve heard of a web debugging proxy, and if they have, chances are they’re thinking of either Fiddler or Charles. While not quite as old as Fiddler, Charles is still widely used, and serves as a good starting example.

The best way to imagine a proxy like Charles is as a net your computer is casting into the flow of requests and responses heading in and out. Every time a session passes through, Charles will add it to its live traffic stream, showing you the URL for the webpage, the method the computer wants to use, the size of the session in bytes, etc. Clicking on a session in the stream will show you even more information, like when the request and response were sent and received, the HTTP headers that hold various metadata, and any code like JSON, HTML, or XML that’s being sent to or from the computer. This is why it’s called a debugging proxy; all the information it provides is useful for analyzing how a website—say, one that you’re currently building yourself—interacts with the rest of the Internet.

A toolbar at the top of the interface gives you even more options. You can throttle your Internet connection to simulate bad WiFi, or repeat a selected request without having to go and reload the website. Two of these actions are very important, however. One lets you compose your own requests by taking an existing one, manually editing it, and then resending it. For someone with a good knowledge of the structure of URLs, this can be a very effective tool for seeing how a website reacts to basically anything you can throw at it. The other option lets you create breakpoints: think of them as little sub-nets on either side of the main net that capture sessions before they even get to Charles, and let you decide whether to let the sessions through, redirect them, or block them entirely.

That’s a pretty good summary of the three things a web debugging proxy should be able to do: show you the inner workings of each session that passes through your computer, send custom requests, and edit server responses. Let’s see how Proxyman and Fiddler handle these tasks.

Proxyman

Unlike Fiddler and Charles, Proxyman is brand-new; it was created by a small team in Singapore and released just this year for Mac. But this isn’t the only reason it’s a good contrast with Charles. Although Charles is cross-platform and undoubtedly effective, it is one of the least user-friendly apps I have seen. In addition to a rather bland and uninspired style for the documentation and UI, the 1-month free trial inserts artificial 5-second delays at random, and will force-quit itself after 30 minutes, meaning actually doing anything substantial with Charles requires shelling out $50.

Proxyman, on the other hand, tries very hard to come off as bouncy and friendly as possible. Its website declares the program “Modern and Delightful,” and its UI provides just as much information as Charles with a lot more color. Being easy on the eyes is good for a computer program, and Proxyman manages to make even giant blocks of code fairly easy to parse quickly. In addition, its free version only limits the number of internal tabs you can have open, preventing you from multitasking but at least letting you use the darn thing.

Proxyman takes Charles’s ability to edit requests one step further by having a mechanism specifically for creating requests from scratch. The “Compose” tool, as it’s called, lets you make your own requests without having to use an existing one as a base. This gives you a lot more freedom as to what exactly you can send, as long as you know the format that the website’s API understands. (API stands for Application Programming Interface; it’s essentially a screen in front of a piece of software that lets it respond to requests from different pieces of software.) However, Proxyman has no equivalent to Charles’s breakpoints, essentially trading response editing for request editing. It’d be cool if there was a program that could do both equally well.

Oh, wait.

Fiddler Everywhere

Having a separate tool for writing custom requests is a cool idea. So cool, in fact, that Fiddler Everywhere has separate tools for writing custom requests and for editing responses. The Composer and Auto Responder tabs, when combined with the Live Traffic Stream, create a three-pronged approach that no other program in the business can match for versatility. You can add headers and parameters to the request with key-value pairs, letting Fiddler do the work of converting them into the proper format for an API call.

The most mind-bending thing about Fiddler, however, is the Auto Responder, which lets you create a conditional ruleset for incoming responses. If a response meets the criteria you lay out, you can specify almost anything you can think of to happen to it automatically. There are so many ways you can fiddle (haha) with webpages; replacing the logo on Google’s homepage with a picture of your choice is only scratching the surface.

And beyond that, Fiddler is just as user-friendly as Proxyman. The only things the free version restricts are the inbuilt collaboration features (oh yeah, you can share sessions, requests, and rules with anyone via email, did I mention that?) and the number of Auto Responder rules you can have active at once. Not only that, but its online documentation is crowdsourced, meaning that it can draw on the insane wealth of knowledge Fiddler’s huge user base has gathered in the 17 years of its existence. An easier explanation of how to use such complex software you will not find.

There are plenty of web debugging programs out there, and they’re all good at different things. But an important thing to note is that they all pick one or two things to focus on. Some are just requests composers. Some are just response editors. Charles and Proxyman have a fleshed out live traffic stream with some functionality in one of the other areas. But even if some of the other programs do one of those things really, really well, Fiddler is the only one that does all three. And it barely sacrifices any depth at all to get that breadth. Sure, if you know you only want to write API calls, it might not be the best choice; but for general applicability, nothing’s better.

Chrome Developer Tools: Differently Useful

Jackson Carter — Wed, 26 Aug 2020 19:40:15 +0000

In the last post, I explained what a web debugging proxy is and what it does, using Fiddler Everywhere as an example. If you missed that, the gist of it is that a web debugging proxy is a program that captures communications between your computer and an Internet server. You can use a proxy to analyze the communications, called “sessions,” as a whole, write your own requests for the computer to send, and modify the responses that the server sends back. Different proxies are good at one of those things or another, and Fiddler is unique in that it can do all three really well. But there’s one question about web debugging proxies that I didn’t mention. It comes up a fair amount in conversations about Fiddler, especially with people with computer science backgrounds, and the answer is fairly complicated.

Isn’t that what Chrome DevTools does?

It’s a very good question, to be sure. Chrome DevTools looks fairly similar to a web debugging proxy like Fiddler, showing a list of sessions and providing detailed information about their contents. Not only that, but it also gives a huge wealth of additional information about the webpage you’re currently viewing, it’s built directly into a very commonly-used browser, and it’s totally free. You don’t even need to download anything, just learn where the button is or what the keyboard shortcut is to bring it up. (It’s Ctrl-Shift-I, or Cmd-Shift-I on Macs.) All of this makes it an incredibly useful and ubiquitous tool for web developers, and you’d think that its existence would make separate proxy programs unnecessary.

However, there is an important difference between what a proxy like Fiddler does and what Chrome DevTools does. To find out what it is, and what it means for which one of them to use, we have to look a little deeper at what, exactly, the data that each of them display means.

What does Chrome DevTools do?

Google Chrome’s Developer Tools, commonly known as Chrome DevTools, is a sidebar that can be brought up on any webpage in Chrome and displays a large amount of under-the-hood information about the webpage. The tools are accessed either through submenus or through the keyboard shortcut Ctrl- or Cmd-Shift-I. The information is spread out across no less than nine different tabs:

• The Elements tab contains the DOM (the browser’s view of the webpage HTML, which can be edited in real-time). Different parts of this document, when moused over, will highlight the parts of the webpage that they encode. This allows insight into how each aspect of the webpage is represented in the code, even for people who don’t know HTML. Clicking on an element of the DOM will show all the CSS classes attached to that element and allow you to edit them. Subtabs of the Elements tab include Styles, Event Listeners, DOM Breakpoints, Properties, and Accessibility.

• The Console tab displays information about any issues the page ran into when it was loading. It also, as the name suggests, provides a JavaScript command-line interface to run commands that can view or alter the JavaScript data for the page.

• The Sources tab displays the folders and subfolders that the page you’re looking at is located in, essentially breaking down the page’s URL and showing you what each of its parts means.

• The Network tab displays all HTTP requests that were made while the page was loading in a similar structure to Fiddler’s live traffic stream. It also has a waterfall chart showing the exact timing of each part of each request down to the microsecond.

• The Performance tab is essentially the expanded version of said waterfall chart, showing multiple graphs of how long each part of the page loading process took.

• The Memory tab lets you take “snapshots” of three different kinds of memory allocations related to the page, letting you know in very specific terms how much memory is being used and where.

• The Application tab lets you inspect all the resources the page has loaded, including databases, cookies, images, fonts, and others.

• The Security tab is fairly self-explanatory, giving information about any security issues on the page.

• Finally, the Lighthouse tab lets you generate a performance report for any website you create.

Whew. That is…a lot of stuff all at once.

What does that mean for Chrome DevTools vs. Fiddler?

The important thing to note here is that the information you can get from Chrome DevTools is not the same as the information Fiddler gives you. It’s a little hard to see because there’s just so much of it, but the Chrome DevTools info is all very specific to the webpage you’re viewing. You can see the DOM for the webpage, the JavaScript for the webpage, the requests the webpage made while it was loading and their timing data, the cookies on the webpage, and so on; but there’s nothing at all about anything besides this particular page. Fiddler, on the other hand, doesn’t show nearly as much detail per webpage, but it isn’t limited to one page at a time. You can see all the Internet traffic for your machine, write requests to any website you can think of, and change around server responses in a whole load of ways. Fiddler is essentially an expanded, broader-scope version of Chrome DevTools’s Network tab.

Because Chrome DevTools essentially operates on a level below Fiddler—the webpage level rather than the computer level—the question of “which one is better” doesn’t apply, because they’re not in competition with each other. They do jobs with a slight but very important difference. Fiddler is good for seeing how a page reacts to outside pressures, like bad requests and jumbled responses, and for monitoring how it interacts with the rest of the Internet. Chrome DevTools, on the other hand, is all about looking inside a page, seeing what makes it tick, and checking if there are any screws loose. They both provide valuable information at different stages of the web development process, which means that they work best when used in tandem. In fact, there are quite a few ways in which they interact directly. For example, one of my favorite things about Fiddler's Auto Responder is that you can set up a rule that replaces a picture on a webpage with a picture of your choice. To construct this rule, you need to find the part of the page’s DOM that contains the image you want to replace, so you can put that code in the conditional part of the rule. Wouldn’t you know it, the DevTools Elements tab has the DOM right there, and shows you directly what parts of the code correspond to what parts of the page. It’s like it was meant to be.

I know it’s tempting to think that there’s one method you can use for everything, but the truth is that the more different answers you have, the more of the bigger picture you’ll see. Debugging a website is like a jigsaw puzzle; Fiddler can only give you some of the pieces. Chrome DevTools might give you some of the same ones, but it’ll give you a bunch of its own that you can’t get anywhere else. Use them both, learn how their answers fit together, and you’d be surprised at what you’ll be able to see.

What is Fiddler Everywhere?

Jackson Carter — Tue, 25 Aug 2020 16:13:01 +0000

Originally written by Eric Lawrence in 2003, while he was working for Microsoft on the Internet Explorer development team, Fiddler is a web debugging proxy; a piece of software that can capture Internet traffic and display and alter its contents. For the past 17 years, Windows-based web developers have used it to test their websites for vulnerabilities and shortcomings. Fiddler Everywhere is a brand-new version of Fiddler that can run on macOS and Linux as well, with updated features for a new era of web development.

The details of what Fiddler Everywhere does are quite complicated, and require a fair amount of knowledge as to how the Internet works under the hood. Let’s start by explaining what exactly a web debugging proxy is.

What is a web debugging proxy?

The key to understanding web debugging proxies is HTTP (HyperText Transfer Protocol), which provides the foundation for the entire internet. (Most of the time there’s an S at the end. The short explanation is that it stands for Secure, which it is. The long one is much more complicated and not really necessary for the basics, so we’ll skip it.)

A protocol is a way for the different computers that make up the Internet to talk to each other. Protocols define certain actions, called methods, that a computer can use to interact with the Internet. GET, for example, is an HTTP method that returns the data of a webpage; POST provides data from the computer to a webpage; and so on. But a computer can’t use these methods just like that, because it doesn’t have the data of any given webpage immediately available. The only computers that can access that data directly are that website’s designated servers. So the computer has to ask one of the servers to carry out the method for it.

The way it does this is by sending an HTTP request; a little packet of data that specifies the computer it came from, what method that computer wants to use, and what part of the website it wants to use it on (and a whole bunch of other stuff). This request zips its way over to a server, which reads it, executes the method, and sends back a response; a similar packet of data that contains anything that the computer needs to display the method’s results (the HTML file for a webpage, for example). Each little back-and-forth between the computer and the server is called a session.

Now, computers can’t just send requests directly to the server; they have to bounce them between various proxies, programs which forward requests and responses along and help them get where they need to go. With some proxies, that’s all they do; others can store, redirect, or even edit and block requests and responses that go through them. Web debugging proxies are the latter kind; they intercept sessions involving the computer they’re installed on and let the user analyze and modify them.

What does Fiddler Everywhere let users do?

Fiddler Everywhere’s features are divided among three main functions; the Live Traffic Stream, the Composer, and the Auto Responder.

The Live Traffic Stream

The Live Traffic Stream is the first thing you see when you boot up Fiddler Everywhere, and is arguably its main function. On the left is the Stream itself, showing every session as it’s captured. Each session displays its host server, its URL, its HTTP method, its HTTP result code (usually 200), the size of its body in bytes, its caching data, its content type, and whether it’s using HTTP or HTTPS. Any one of these properties can be used to sort or filter the stream. The stream can capture HTTPS traffic, but you have to enable it in the settings menu. Doing so requires trusting Fiddler’s root certificate, due to the way HTTPS works. Right-clicking on selected sessions in the stream will let you copy them, comment on them, save them, share them with other users, or use their URL to create a Composer request or an Auto Responder rule.

To the right of the stream are the Inspector windows, letting you see information inside both the request and response simultaneously; request on the top, response on the bottom. Multiple tabs let you see different parts of each:

• The Headers tab (for both) shows various metadata assigned to the request/response. HTTP headers are very specific and standardized, and learning what each of them means can help you understand what any session is trying to do without even looking at the rest of the tabs.

• The Text tab (for both) shows any text that the request/response contains.

• The Web Forms tab (for request only) shows HTML form-data represented as a set of key-value pairs.

• The Image tab (for response only) displays any images that the response contains.

• The Web tab (for response only) gives an approximation of what the data contained in the response looks like when rendered as a webpage.

• The Cookies tab (for both) shows a message detailing any cookie data that was sent or modified.

• The Raw tab (for both) shows as much of the data as possible at once: headers, then text, then code.

• The JSON tab (for both) displays any JSON (JavaScript Object Notation) code contained in the request/response. JSON is a format for storing complex data in key-value pairs.

• The XML tab (for both) displays any XML (eXtensible Markup Language) code contained in the request/response. XML is a language defining a document format that both humans and computers can read without needing a compiler.

The Live Traffic Stream can show you the cause of many issues that you can run into with a webpage. Maybe it’s taking a long time to load, or parts of it are refusing to: filter the sessions down to just the ones for that website, and you can see if there’s too many requests, a request or two that’s too big, or one that’s failing to go through. The stream’s controllable specificity means it can even be helpful in casual settings, not just when you’re trying to build a website.

The Composer

The Composer is a separate tab that allows you to send your own requests directly without going through a browser. The bar at the top is for the URL you want to request; you can also set the HTTP method and even version of HTTP that you want to use. Below that bar is a window showing your request; like the Inspectors, it has multiple tabs with different information, all of which you can edit.

You can add headers and parameters in their respective tabs; enter them in a key-value format and they’ll show up properly integrated into the URL above. In adding headers and parameters, you’re modifying the call to the website's API (Application Programming Interface), which is a system that lets two different pieces of software talk to each other. There’s also a Body tab that lets you add text, and a Raw tab that shows you the request in its entirety. Once you send the request, the response it generates will show up in the box at the bottom.

The Composer is an incredibly useful tool for seeing how a website responds to all manner of requests, valid and invalid. It can provide valuable information on a site’s vulnerabilities. In addition, it can serve as an introduction to API development, a field related to web debugging that is highly complex on its own.

The Auto Responder

The Auto Responder might be Fiddler Everywhere’s most interesting feature. It allows you to create conditional rules that can alter the content of server responses. For instance, you can write a rule that activates when any page on a certain site is requested, that redirects the browser to a different page. Or delays the response by a certain number of milliseconds. Or edits the headers or parameters of the response. Or blocks the response entirely. The Auto Responder can do almost anything you can think of (you can even use regular expressions in the conditions); the only limit is what you know how to translate into the correct format.

Like the Composer, the Auto Responder lets you probe a site for weak points in places you might not expect. Can the site handle multiple redirects in a row? Does replacing this picture with another break the formatting? How well does the site hold up when the connection is slow? The Auto Responder also allows you to either group rules together or set a priority order, allowing for even more complex testing by letting the rules interact with each other.

Collaboration Features and Fiddler Everywhere PRO
By default, Fiddler Everywhere is totally free, and all the features described above are available in full. However, the PRO version, which costs either $12 a month or $120 a year, has greatly expanded collaboration features that make it much easier to work together with other users.

The free version of Fiddler Everywhere has some of these features, but in a reduced form. It is possible to save sessions and share them with other users via email, but only 5 at a time, and with only one other user. In addition, the Auto Responder can only have five rules active at once. With the PRO version, however, these caps are eliminated, as well as the cap on how big shared sessions can be. (In both versions, it is possible to share Composer requests and Auto Responder rules as well.)

With PRO, you can be part of a larger team all working on the same project. But without PRO, you can still do all of the fundamental parts of web debugging—capturing traffic, writing requests, and modifying responses—to the exact same extent. Fiddler Everywhere is one of the only products I can think of where the free version is just as viable as the paid version. Sure, PRO is “better” by virtue of having fewer restrictions, but those restrictions are only on things that, if you’re working alone, you might not even use.

Very few web debugging proxies manage to have the same breadth of usage as Fiddler Everywhere. Its competitors focus on one or two of the three parts and disregard the others. Fiddler Everywhere is the only one of them that can do everything, and barely sacrifices any of their depth. It’s a wonderfully well-rounded program that is perfect for any situation or level of experience. Download it right here if you want to give it a shot.