Let's talk about webdrivers

The internet

The connections between your computer and a server (Google server, Amazon server) any of those is based on HTTP (or HTTPS) requests and "code" files as Html, CSS, and Javascript. As a client-server connection, your computer has a client software installed, wich can be Google Chrome, Firefox or any other. This software manage the communication between the client and the server.

So we can organize in this way: You (Web Browser) requests Server (online service) using HTTP(S) as communication protocol. When the files are delivered by the server the browser can design the web site and it's functions can be load.

Ok, this was a basic review.

A webdriver

We can now work a little further on the client's interpretations.

A webdriver pretends to be a web client making requests to the server as if it were a normal user. This way the software can access the Html of each page, capturing informations when needed.

I can give you usage examples, I'm from Brazil, and here we have some startups, and big companies that use this kind of system everyday, financial industry uses to capture information about their client's money situation, other companies can use to check company's money situation in government web sites.

How does it work

Inside the HTTP or HTTPS request we have the CRUD operations (Create, Read, Update Delete). In the HTTP it's called GET (Read), POST (Create), PUT (Update), DELETE (Delete). This is the first thing to know.

The HTTP request carries a body and a header, when it's a POST or a PUT request, it's expected to have a body content. When the request is a GET or DELETE, it usually does not have a body content, just headers and the URL.

We know that we need to pretend to be a browser, so we can receive the information that we want. But how will we know wich request and when we have to do? Usually you have to use a "sniffer" to track every request sent by your web client and then do the exactly same requests.

I like to use "Fiddler" (https://www.telerik.com/fiddler) it's very simple to use, you can just target the web client and track the requests of any navigation you want. Of course you have to target the web client and access the web site, the exactly path you want the webdriver to process.

The hardest part will be understand wich request is done, when and why, and of course where are the variables of each request.

Architecture

Everything is a state machine, your webdriver couldn't be different. I will show a model of architecture, wich can be good or not for your usage, but I have used many times this architecture in my webdrivers.

This is a really simple architecture that will be represented without the fixation of a specific language.

Goals:

-Map all the pages of the navigation.
-Parameters for each request and each page.
-Cookies (Parameters and navigation).
-Identify the informations of interest in each page (Regex).

So in your code you will need...

Enum (example):

    -Page 1.
    -Page 2.
    -Page 3.
    -Page 4.

Extract (example)

    -Url
    -Dictionary<string, string> query
    -List<string> regexs
    -Dictionary<string, string> header
    -object body

Dictionary <Enum, Extract> (example):

    -Page 1, Extract.
    -Page 2, Extract.
    -Page 3, Extract.
    -Page 4, Extract.

And the main function (example):

-Loop (navigation.situation != End):

    -Execute the functions NavigateTo(currentPage).
    -Get the response content (Html, JS and CSS).
    -Execute the regex of the page.
    -Get the parameters (including cookies, and Html         
    variables) for the next page navigation.

-End Loop

And for the end the extract module, wich can be done in markup languages, databases, tables, and whatever you would want to.

This was a introduction to webdrivers and it's architecture, of course it's a complex subject, and this is not even a half of it. What is usually done is make frameworks to deal with Html readings (the Regex for example), and even a automation of the navigation. Using XML tags to describe regex, urls, query url, and the others properties, it can just be uploaded to a generic webdriver, wich will just use the XML uploaded to navigate whatever web site you need.