DEV Community: David Burns

Flakiness isn't from your test framework

David Burns — Tue, 28 May 2024 10:18:20 +0000

The other week I saw Filip Hric share a post from Gleb Bahmutov, ex-principal engineer for Cypress, explaining that the way cypress works and not using transport layers like playwright or WebDriver based frameworks makes its tests less flaky.

That’s 100% not the reason your tests are flaky. I’m not going to lie, this shocked me that Gleb would say this as I’ve always thought of him as a good engineer after seeing his work on Cypress. The reason your tests are flaky is more down to how you interpret the UI and how the browser runs the code, or to put in a different way, you’re thinking about your synchronous steps of a test while they’re running in an asynchronous way. This mismatch leads to flakiness.

Now add in different browsers interpreting the code in different ways leads to more flakiness. This is why some frameworks don’t want to or can’t support different browsers.

Single threadedness of JavaScript

Cypress runs in the page that’s being tested. That means Cypress is hemmed in by the same-origin policy. It injects what it needs to the page. The problem is this means CPU has to swap in commands from different tasks. This could lead to less flakiness but it’s because it’s running much slower. Since there is no guarantee of ordering of commands between tests and the front end, the reduced flakiness is pure chance.

Selenium, when Jason Huggins created it, used this technique for automating the browser and Selenium moved away from it when we merged with WebDriver. Hugs have been calling this out for ever. It’s also the reason why you can’t do basic things like trusted events, iframes, or navigating between different origins.

// Detect dark theme var iframe = document.getElementById('tweet-1177619507927490560-348'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1177619507927490560&theme=dark" }

Driving the browser from inside to outside, where outside is your webpage, is always going to give you a more realistic testing experience.

Transport layers

The transport layer for speaking to the browser doesn’t affect the flakiness. Its main benefit is scalability. If you need your tests to run in the same browser as the runner then you struggle to scale. Since Selenium’s main transport system is based off HTTP we know it’s highly scalable. Cypress tests are less scalable because it wanted to do everything on the browser.

but what about CDP?

What about it? It’s a chromium based protocol. Playwright uses it, Puppeteer uses it, and this might shock you, Selenium uses it. This is firstly how Edgedriver and chromedriver speak to the browser. So if you’ve used chromedriver at any point in the last 8+ years you’ve used CDP. Selenium can also speaks directly to the browser using CDP for some commands. We see this with their network intercept API and logging APIs are examples. Now, webdriverio also supports WebDriver and puppeteer through these APIs. So if we follow Gleb’s post, we need to move everything down to playwright/puppeteer part of his diagram.

The one downside to having to rely on CDP is you limited to chromium specific APIs. These are not stable by design which the Chromium team will tell you. It’s the reason why chromedriver/edgedriver needs to be updated with each browser release. Fortunately, Selenium Manager can auto update your drivers for you without you needing to worry. If you're not using Selenium Manager you will have to update all your dependencies which you would have to do with Playwright or puppeteer. If you’re using the Selenium event driven APIs then you will have to update your selenium dependency.

As mentioned, it does limit us to chromium specific APIs which is why Selenium is working with Google, Apple, Mozilla, and a few other little companies to bring about the new Webdriver-bidi spec. When WebDriver-BiDi is out to all browsers then the requirements for updating with the browser will drop like they did with Firefox and geckodriver.

The puppeteer team is supporting WebDriver-BiDi. A debug protocol is not the best for automation as it relies heavily on browser state. It’s great for debugging but not automation. Since this work is happening in the open, we are happy for the playwright and cypress team to come collaborate with us.

How do we solve flakiness then?

Auto-waiting, minimizing what your tests are doing, and root cause analysis of failures. Webdriver.io, Nightwatch, playwright, puppeteer, and cypress have the auto waiting. All at different levels. There is some opinions that differ on what should be waited for but they are aiming for the same end result.

Selenium has had auto waiting in a minimal wait with the explicit and implicit waits. People can struggle with them. These waits are opinionated like the above. So... we're down to which opinions we like. So... if you're learning a tool for a CV... well it's just an opinion that's different. How to do web testing, and understanding it, that is the super power you need.

As an aside, when I was at Mozilla, automattic came to us saying they wanted puppeteer support as they were dropping selenium because of flakiness... and then then the flakiness was still happening with puppeteer

Flakiness has been around for a long time. Simon Stewart wrote a good blog post about this problem 15 years ago and how to solve some of them. This is not a new problem and someone telling you their framework will solve it is lying.

Unfortunately, there will still be some flakiness… and it’s not your fault.

Front end frameworks hate testers

These frameworks make flakiness equal when it comes to testing.

Browsers have had to put a lot of effort to improving the speed of rendering and painting to handle these because of the constant moving in and out of the DOM. These asynchronous changes to the DOM the way your tests run. In selenium you’ll hit a StaleElementReference. These are painful but auto waiting can solve it for you easily. Batteries included systems, like NightwatchJS, webdriverio, playwright, and cypress try make how this works opaque to the end user.

Asynchronocity is hard

As I alluded to above, we don’t think of tests in the same way we think of rendering front end tests. When we’re wanting to take data off a server and render it, it’s all happening asynchronously due to how JavaScript and fetch APIs work. The move to server side rendering again is not going to solve this either… it’s just moving the heavy lifting around. Yes with promises we can write code that looks synchronous but that’s just for where that bit of code is. A slow response from a server can impact our tests. Having your tests and front end competing in the same thread won’t make your tests less flaky other than by sheer luck.

Finally… work in the same world as your users

So is there any benefit to running your tests in Electron? Well… how many of your users use your site with electron? Probably the same amount as would use playwright-Firefox or playwright-WebKit. It’s a number that is very close to 0.

I’ve talked about this many times about having your tests work. It’s important to have a number of tests running in the environments where our users are. I’ve got examples in my talk from last year. Different environments, like mobile versus desktop, can also lead to different reasons for flakiness.

So… when picking a framework, pick one that works well for you. It should be able to do a Login Test easily. If it can’t then it shouldn’t be used.

Engineering culture is hard to make but easy to destroy

David Burns — Mon, 21 Nov 2022 11:58:41 +0000

Like most people, I have been watching, with utter despair, the downfall of Twitter. Twitter, the community, has been a bit of a cesspool in general but there are some really great micro communities that help each other.

I know I have met amazing people on there and from there cultivated great friendships.

The one thing that has shocked me with Elon Musk, is not how he said he was going to down size the company. He told people that at the beginning. It's the callousness of all his actions!

Bullying is never ok

Anyone who has seen the news reports from Union Busting at Tesla to issues with Self-driving Cars(now a class action lawsuit) to him seemingly breaking WARN Act in California, know Musk has little regard for laws around him.

This has lead to, at least if rumours are true and what we're seeing at Twitter suggests they are, a constant bad engineering culture within all Musk companies. Bullying and Harassment.

Now, at this point I can foresee Musk supporters thinking "Well you're not a billionaire so why should we listen to you?". You don't need to listen/read anything I put out there. No doubt that Musk has been a shrewd investor, but the concepts and successes are built on the people doing the hard work. If my family had also been able to exploit South Africans during Apartheid, I too would be significantly wealthy.

But being a shrewd investor doesn't mean it's ok to bully or harass folks.

Building a healthy culture

A healthy culture is a productive culture in the same way a healthy population will be a prosperous nation. When people feel they can make mistakes safely they are going to learn and build and grow with you. It's one of the main reasons in the tech world we have moved to blameless retrospectives.

One thing that is true, and underrated, is "Every hire will impact your culture". If you go for a group of Chads, or mix it up with people from different backgrounds, you will have a different culture. I have worked on teams where there were 6 of us (3 David's and literally no women). I learned more and grew more as an engineering leader by having different people around me.

Words matter

Words matter, and the more multinational a company is, the more important words can be. How you speak to people is going to show them how they can interact with you. I know when I speak to certain parts of the world I need to make sure that if someone calls me sir that I remind them that I am not knighted and they should just call me David.

Why? Because if they fail they are likely to feel that they can't speak to me. I also remind them that fail stands for First Attempt In Learning. Calling people by their prefered name, and definitely not their Dead Name, means they are likely to listen to you. It's really the little things.

Leaders need to be at the front

Leaders showing people how to do things from the front, even if it makes you feel "weak", creates empathy between you and your team. It's long known that facism is beaten by empathy. Empathy is not a weakness, it's a super power. It also will help them feel like they can challenge you if you are wrong. Musk has shown that if you challenge him you will get fired. He is now surrounded by sycophants. People won't challenge him and it's leading to him looking like the emporer with his new clothes.

Culture can be destroyed in an instant

One thing that I learned back in my Mozilla days was that culture can be destroyed by poorly chosen words by leaders or
poorly chosen actions. The actions of leaders can hurt people. If you want amazing products you need diverse teams. It's important to It takes time to build a culture that people want to be part of, that allows failure and learning. Trust can takes weeks to months to years to craft. Ask any manager who does 1:1s...

Building a rapport that allows people to be their best within a team so you can have a group of people at their best forming the best team takes time. You can't just force culture on people, especially since each person can change the culture in the group. It needs to get there through discussion and tradeoffs.

It takes time to build world class engineering and a good set of leaders who can handle the tradeoffs that teams need. Build out what we need when we need it. Musk clearly doesn't have that skill. I bet there are a couple good managers at his companies that can help deliver things but it's clear from his actions it's not him.

It's going to impact other companies. Twitter, Telsa, and SpaceX will feel it soon in their recruitment as that dries up quickly with people refusing to work there because of the actions of the CEO. I definitely wouldn't want to work there!

Finally

And finally, you need to read Chesterton's Fence and think about all the things that Musk is doing. It will hopefully frame why a lot of well respected people in their field are utterly shocked by the actions to people and systems happening in Twitter!

I am on Twitch - Come work on Selenium with me!

David Burns — Tue, 26 Jan 2021 08:36:34 +0000

At the end of 2020, I wanted to try something new, well new for the Selenium community. I created a new Twitch channel.

Why?

The main reason is that I wanted to show people that working on Selenium is not scary. You don't need to be a "rockstar" engineer. Actually, Rockstars are actively discouraged since they normally just trash hotel rooms.

What have I done so far?

I have shown people around the codebase. With Selenium being a monorepo which we do Trunk Based Development. This can seem scary to some people seeing the code for the first time. I showed that it's not scary!

I also showed people how they can contribute to the Selenium Documentation. Documentation is how a number of Selenium Core contributors got into the project as it is a definite pay it forward system, making sure that future Selenium users
have correct working examples.

And last week was about starting to remove all the python 2 specific code from the python part of the tree. This is not complete and, this week, I hope to work on it a bit more. I will be leaving the code base littered with TODO if anyone wanted to contribute.

When

At the moment there is a session every Thursday at 2 PM GMT. Hopefully, as the UK comes out of lockdown, I will be able to add a few more sessions during a week. If you subscribe to the channel you will be updated as sessions change and be notified when a session has started. So please do subscribe! If you miss a session they are stored on Twitch for a couple weeks so you can always catch up later.

Future Sessions

As we go through the year if you have ideas of things you want to see in a session, please let me know. My DMs on Twitter are open or feel free to add something to a session chat.

Originally posted on my blog

Browser Testing and Tools WG Meeting @ TPAC 2020

David Burns — Mon, 02 Nov 2020 12:03:38 +0000

It's that time of the year where working groups from the W3C meet up to discuss the various standards that
are being worked on.

Within the Browser Testing and Tools Working Group, there are 2 different standards.

We have:

The first is what is commonly supported by the Selenium Project and has support from Apple, Mozilla, Microsoft, and Google in their browsers. It is also supported by various Selenium in the cloud providers like Sauce Labs and BrowserStack.

As the world has moved on we have felt the need to add new APIs and move Selenium to be more event-driven. This is where we are learning, and collaborating, with projects like Puppeteer to make sure that we can improve the Browser Automation space. This is where the WebDriver-Bidi Specification comes in. It has broad support from the browser vendors so you can use official browsers and not be limited by the JavaScript sandbox. Some of the newer frameworks can't guarantee that.

If you're curious about we discussed this week, feel free to read it up on the W3 Wiki.

How Selenium Works: Episode 7 - Driver Executables

David Burns — Fri, 08 May 2020 10:52:09 +0000

So this episode is going to be slightly different to previous episodes as this question came up this week. Why do browser vendors and the Selenium Project ship a driver executable? And Why do I need to update it with every browser release?

Why do I need to download something?

This is one of the biggest pain points I have seen people struggle with when using Selenium. The majority of the people who struggle are those who do not have a lot of experience programming. So why do we download this executable?

In Episode 1 I mentioned Selenium under the hood speaks a REST-ish API.

The driver you are downloading is an HTTP server for the client libraries to be able to speak to.

So... Why not just get the client bindings to speak whatever is native to the browser?

Unfortunately, as I mentioned in Episode 1, doing that increases the amount of work each binding maintainer needs to do and potentially limits who can write the bindings.

It also means any setup required for the browser needs to be done by the bindings. If you're using a Chromium-based browser or Firefox, the bindings will need to know where to find the browser executable normally, as well as make sure the profile directory is created properly. It also puts the onus of fixing any breaking changes the browser vendors do to their startup setups on to each of the binding maintainers.

Trust me, the Selenium Project used to do that and every browser release day meant a mad scramble. We would start running tests in an attempt to make sure nothing was broken. We would fix the problems and then release. This herculean effort by core contributors meant we were limited in adding new features.

Not having to scramble and making sure it was simple for client binding maintainers to keep up to date was a great push for creating the WebDriver Specification.

What about automatic downloads from the Selenium Project?

Well... This is something we have wanted to implement in the past but we haven't had time. Fortunately, there have been other projects that have come along and made this work easier. In the future, we are likely to try to incorporate one of these projects or we might look to see about implementing something ourselves.

This will be something we will try look into once we have shipped Selenium 4.

How Selenium Works: Episode 6 - sendKeys

David Burns — Mon, 20 Apr 2020 08:12:06 +0000

After an interaction on the last weekend of January 2020, on a Selenium Issue where someone said “why can’t you just…” after I explained the issue I thought that I would start explaining commands in Selenium WebDriver and why we landed on the design that we have today.

I will repeat this on every page of the series but a lot, an annoying amount sometimes, of thinking goes into how every little bit of Selenium works.

In this episode, we are going to look at what goes into clicking. For simplicity, we are only going to be looking at the work that goes into doing element.send_key("a string").

You have an element you want to send some text, now what?

So you have found an element and you need to send text to it. Selenium will go through the following steps before it even sends any text.

Check Element is still on the page.

When we were finding an element the browser would have returned a representation of that element. All subsequent calls to that element need to be checked that the element is still on the page. If it is not you will get a StaleElementReferenceException.

Check the element is visible

Once we know the element is still on the page we need to make sure that the element is displayed. This is important as we would never expect a user to click on an element that is not visible. Since a user would never click there we need to make sure that Selenium can't click here too. If it is not displayed you will receive an ElementNotVisibleException.

Scroll to the element

Next, we need to make sure that we can scroll to the element. To have all the events fire properly we are going to need to have the element in the viewport. If the element is not in the viewport after trying to scroll to it, the browser will return an ElementNotInteractableException.

Check if the element is interactable

The browser will now check if the element that will receive our text can receive it. For example, we will look if the element might be disabled. If it looks like the element can't receive the text, the browser will throw an ElementNotInteractableException.

Process Text String

The next step is we need to process the string that has come in and split it according to grapheme text clusters. We do this as Selenium has assigned some values in the unicode private use area or PUA for some of the keys. Below is a sample

    TAB = '\ue004'
    CLEAR = '\ue005'
    RETURN = '\ue006'
    ENTER = '\ue007'
    SHIFT = '\ue008'

This allows people to send through combinations of keys like below.

    from selenium.webdriver.common.keys import Keys
    element.send_keys("I" + Keys.SPACE + "love " + Keys.SHIFT + "cheese")
    # This writes "I love CHEESE"

Firing some events off

Now we're ready to fire off some events. The browser will have gotten the string, so in the case above it would have received I\ue00dlove \ue008cheese. The browser will split this into an array and process each character. When it hits one of the PUA it will have logic that needs to apply. For example, with the Keys.SHIFT it will set the shift property on characters that follow.

It will fire off a list of events into the event loop so that it will come across as a trusted event. And when done, it will return to the Selenium script.

What about navigation?

We need to tell the browser to watch for navigation happening thanks to the click. If a click does cause a navigation we apply the same logic if we were navigating.

Exceptions

File Inputs

There is one element that will be an exception and that is the <input type=file> element. When this element is being interacted with the browser will try to check if the string being passed in is a path to a file. If it is not, the browser will error and pass that back to the Selenium script. When using Selenium Grid, Selenium will encode the file you want to upload as a base64 string, send it through the grid, and then let the grid update the file path that will be used.

Selenium will also try to ignore some of the visibility checks. This is because File inputs can't be styled so people do CSS tricks to have a nicer looking element on the page and hide the actual <input type=file>.

IME Keyboards

Send keys do not engage the IME keyboards on the operating system. Selenium assumes a US Keyboard for everything. Fortunately, Send Keys does allow you to send through Unicode characters if you need to type kanji or other languages that would require an IME keyboard. You might not get any special events from the IME keyboard.

How Selenium Works: Episode 5 - clicking

David Burns — Tue, 31 Mar 2020 08:25:06 +0000

I will repeat this on every page of the series but a lot, an annoying amount sometimes, of thinking goes into how every little bit of Selenium works.

In this episode, we are going to look at what goes into clicking. For simplicity, we are only going to be looking at the work that goes into doing element.click().

You have an element you want to click, now what?

So you have found an element and you need to click on it. Selenium will go through the following steps before it even does the click.

Check Element is still on the page.

Check the element is visible

Scroll to the element

Check if the element is interactable

The browser will now check if the element that will receive clicks in the area that we're telling it is the same element as we have passed in. If it looks like it's not the element that will receive the click the browser will throw a ElementClickInterceptedException.

Firing some events off

Now we're ready to fire off some events. We have a special case for <option> to make sure that it's parent element that receives the events.

We then calculate the centre of the element and start sending the events. You can read which events we send in the WebDriver Specification.

What about navigation?

We need to tell the browser to watch for navigation happening thanks to the click. If a click does cause a navigation we apply the same logic if we were navigating.

How Selenium Works: Episode 4 - Finding Elements

David Burns — Tue, 17 Mar 2020 09:17:57 +0000

I will repeat this on every page of the series but a lot, an annoying amount sometimes, of thinking goes into how every little bit of Selenium works.

In this episode we are going to look at how findElement works. We need to be able to find elements before we can interact with them.

Finding an element is easy right?

Well, yes if people designed their applications for testability over anything else! Unfortunately we live in a world where most of the time people think about quality and testing as an after thought.

Anyway... so let's look at the different ways we can look them up.

Search Types

This part is simple, Selenium offers the same methods for searching as you can find when interacting with the DOM with JavaScript. If we can look up things via document.querySelectorAll(aQuery); then it will be searchable via find_element or find_elements. This is searchable from the WebDriver object or from the WebElement object. If you do element.find_element(...) that is equalivent to doing element.querySelectorAll(aQuery) which sets the starting point, or the root, for searching from that element.

Below is an example of how to use find_element:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.get("https://www.theautomatedtester.co.uk")
driver.find_element(By.ID, "someID")
driver.find_element(By.CSS_SELECTOR, "#someID")
driver.find_elements(By.CSS_SELECTOR, "#someID")
driver.find_element(By.NAME, "someName")
driver.find_elements(By.NAME, "someName")
driver.find_element(By.CLASS_NAME, "someName")
driver.find_elements(By.CLASS_NAME, "someName")
driver.find_element(By.TAG_NAME, "someName")
driver.find_elements(By.TAG_NAME, "someName")

We can also look up elements on the page by XPATH.

We can do the following

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.get("https://www.theautomatedtester.co.uk")
driver.find_element(By.XPATH, "//li")

We can also look up links based on the visible text. Visible text uses the }}">isDisplayed algorithm to work out what people can and can't read. Selenium will collect all the <a></a> tags on the page, get the visible text, and then find the first element that has that text.

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.get("https://www.theautomatedtester.co.uk")
driver.find_element(By.LINK_TEXT, "some text")
driver.find_element(By.PARTIAL_LINK_TEXT, "text")

What happens when it finds the element?

When we are able to find an element we keep track of it in a Map. The Map will contain a representation of a element using uuid. The map allows us to look up elements when we get them back from a Selenium test and make sure that element is actually connected to the DOM. If the element is not connected to the DOM and you try use it you will get a StaleElementReferenceException.

and when it doesn't find the element or elements

There are 2 things that can happen.

If you look for only 1 element you will get an exception like:

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.get("https://www.theautomatedtester.co.uk")
try:
    driver.find_element(By.LINK_TEXT, "some text")
except NoSuchElementException:
    pass

If you are looking for more than 1 element you will get an empty list:

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By

driver = webdriver.Firefox()
driver.get("https://www.theautomatedtester.co.uk")
elements = driver.find_elements(By.CSS_SELECTOR, "myShadowElements")

assert len(elements) == 0

Coming Soon: Relative locators

These allow you to search for elements on the page by their relative position to another. For example you can find the element that is above and to the left of another element. See below

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.relative_locator import with_tag_name

driver = webdriver.Firefox()
driver.get("https://www.theautomatedtester.co.uk")
elements = driver.find_elements(
                    with_tag_name("td").above(
                    driver.find_element(By.ID, "center"))\
                    .to_right_of(driver.find_element(By.ID, "second")))

How Selenium Works: Episode 3 - isDisplayed

David Burns — Wed, 11 Mar 2020 15:13:12 +0000

I will repeat this on every page of the series but a lot, an annoying amount sometimes, of thinking goes into how every little bit of Selenium works.

In this episode we are going to look at the vast amount of work that goes into isDisplayed(). This work is used in interaction commands so it is good to get an understanding of how this all works and how other commands use it. From a technical point of view, this is my favourite command and how it works.

Why do we care if it is "displayed"?

So, let's start with understanding why we would even want to care if the element is displayed or not.

Firstly, testing is one the biggest use cases for Selenium so when people do an interaction on the page. Unfortunately, web pages are designed for showing information and not necessarily designed for interactive web pages. As we move to a more interactive web we needed a tool that could at least try workout if elements are visible.

There are two main cases that we will try see if an element is visible. When we call the element.is_displayed() for looking at an element and when we do interactions.

NOTE

Selenium has two type of commands when it comes to interactions. The methods found on WebElement are of the "do what I mean" style: selenium tries to do make what you intended to happen when you type or click. The methods associated with Actions are of the "do what I say" style --- these commands will do exactly what you tell them to, without attempting to interpret what you're actually trying to do.

How does it work?

First of all, we need to understand that element.is_displayed() needs to work without having to make the page scroll. This is important as we don't want to be moving the page unnecessarily.

From there we need to have a look at the element. We need to start looking at the CSS that is on the element.

First, let's see if the element would still be part of the accessibility tree. We don't look in the accessibility tree, we just look at some scenarios where they won't be there and they won't influence any positioning of elements on the page. We don't look in the accessibility tree as this can be an extremely expensive operation in the browser.

An easy scenario like that is when there is display: none on a element like below.

#idOfElement {
    display: none;
}

This will result in nothing being added to the accessibility tree and won't influence the positioning of elements.

Next we need to look and see if we can actually scroll to the element if that is necessary. We are just going to test it as we don't want to actually scroll. This is important for elements that have been transformed to somewhere else on the page. This means that we can't always rely on looking at the DOM tree. I discussed this in a }}">previous post in 2013.

From there we need to do a few checks on items like <input>, <option>, <optgroup> and <map>. <option> and <optgroup> items are "hidden" unless you click on them we assume they are visible... well mostly.

Then we move on to checking the size of the element. For elements that have a 0 size we say they are not visible. We also check their opacity. If the opacity is 0 then we don't view the element as visible.

Once we have done these tests we need then recursively walk the DOM and redo all the tests until we reach the documentElement.

Why the recursion?

Well unfortunately due to CSS we don't always know at what level of the DOM. We need to check each node along the way back. It does mean that isDisplayed can be a little slow to run but it gives us a good approximation if the element would be visible.

Can't we just ask the browser if an element is visible?

Unfortunately no. Browsers when working out what to render they build up display lists and then send these over to the window manager for it to do all the heavy lifting. Browsers have in the past never worried too much about how efficiently the display list was generated. The window manager will try make the display list more efficient and then render. Great!

Now, the thing with display lists is they only tell the window manager what to render for the viewport. The viewport is the area of the page that you can see when you are using the browser. We want isDisplayed() to tell us if an element will be displayed.

We also get into the realms of partially obscured elements. How do we know if an element is truly visible or just a small part is visible. Here we could use some web APIs like document.elementFromPoint(x, y);. Again, this would only tell us about elements that are in the viewport and then there are cases where it an element might be covering another but clicks go to the "hidden element".

I go into this in a little more detail in my Selenium Conf London 2016 talk. You can watch the video.

Unfortunately, no one wants to specify how this would work in a browser so until then we need to have our own way of doing these calculations.

How Selenium Works: Episode 2 - Navigation

David Burns — Mon, 02 Mar 2020 22:56:42 +0000

I will repeat this on every page of the series but a lot, an annoying amount sometimes, of thinking goes into how every little bit of Selenium works.

In this episode we are going to look into the sheer amount of work that goes into navigation.

from selenium import webdriver

driver = webdriver.Firefox()
driver.get("https://www.theautomatedtester.co.uk")

We can see above... it looks easy enough right... no!

Actually, it leads me to a pet peeve in interviews. If someone ever asks you to ever describe what happens in a browser when you type a URL and press enter there is a high probability that they have no real idea what goes into navigation. Anyway... back to Selenium and how we navigate.

`driver.get`

Once we send the request over the transportation layer the real fun starts. We need to figure out where we want to go and tell the browser to start going there for us. Unfortunately this will lead to the first of the many problems for automators.

Certificates

If you have ever worked in major corporate companies where technology is not the core then you will understand the pain that a development team might have to endure. Now double that pain and you're starting to get an idea of the pain that testers and automators have to deal with.

One of these pains is certificates. Companies are cheap and will do self-signed certificates and other monstrosities. Especially in the early days of Selenium when there wasn't services like Let's Encrypt. And even now, most Developers and QA teams rarely have access to change configurations on their test environments or on their CI servers. We needed to come up with a way to circumvent the certificates. (This is one of the first reasons why Selenium is seen as a security risk)

So each of the browsers will allow the poorly configured certificates through when we automate. It's because if we didn't a lot of testers/developers would never be able to test their sites.

Now... that we go round the first problem we need to move on to getting the page to load.

Loading

Once we have got round the certificates we get the page to load. Luckily we don't need to do anything more than the equivalent of

location = "https://www.theautomatedtester.co.uk"; 
// or
window.location.href = "https://www.theautomatedtester.co.uk";

Done...

When Selenium has "finished" a command it will return. So, we just need to wait for the page to be finished loading. To be honest, what does "finished loading" even mean.

In the browser it will fire a number of different events. We will know when the page has been shown and then tell you what it's readyState. Selenium will check all of that and it will also wait for the DOMContentLoaded.

Then there is the problem if you're on the page and you try navigate to an anchor on the page. Let's look at the following example.

from selenium import webdriver

driver = webdriver.Firefox()
driver.get("https://www.theautomatedtester.co.uk")
driver.get("https://www.theautomatedtester.co.uk#someAnchor")

Oh look, none of the page loading events are going to be fired! Yay? Stupid browser is being efficient and will just scroll to the anchor. This means that we can't solely depend on events coming out of the page.

No, so we need to have specialised code for this case and what "done" means in this case. If we get this wrong it's going to create a lot of unstable tests. Unstable tests make people grumpy and we don't want grumpy developers, there are enough of them in the world without unstable tests.

Once we have done those checks, and you can manipulate how we look at those events, if you want it to load quicker, using the Page Load Strategies. This is more of a power user feature so I wouldn't worry about them now but they affect how quickly the navigation command

What about JavaScript Frameworks and navigation

This is where there is "fun". A lot of frameworks will do a lot of loading after their initial load on to the page. If you have ever worked on a single page app, or have ever used a single page app, you will have seen a lot of items render as the data comes in. Unfortunately it does mean that you can't just rely on the navigation command returning. You will have to add a WebDriverWait command to your code, as shown below, to make sure your test is in the right state before going off to do what it needs to.

from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait

driver = webdriver.Firefox()
driver.get("https://www.theautomatedtester.co.uk")
element = WebDriverWait(driver, 10).until(lambda x: x.find_element_by_id(“someId”))

Conclusion

When loading a page, don't always rely on Selenium returning when the page has finished. If you need to look at an element on the page then do that. Just be aware of how the JavaScript on the page could mutate the page after the initial downloads are done.

How Selenium Works: Episode 1 - Transportation

David Burns — Fri, 28 Feb 2020 09:06:23 +0000

I will repeat this on every page of the series but a lot, an annoying amount sometimes, of thinking goes into how every little bit of Selenium works.

Why?

Selenium, by chance and being good at what it does, is used by millions of people around the world as well. It's how companies from Microsoft and Google to the small startup make sure that their site works in every browser.

How Does Selenium talk to the browser?

Selenium, over the years, decided that we were going to use HTTP to speak to the browser. We built a REST-ish API that every client binding could use and hopefully get the same results.

HTTP and REST-ish? Really?

Yea...

Let's start with the HTTP part. When we started we had to have a unique way to speak for each browser based on the best way to speak to them. So for Internet Explorer we wrote COM code. It was fine, it worked but gave us nightmares. For Firefox we wrote a monstrocity that read line by line and, thankfully, due to Mozilla's "make the browser yours" attitude we could do a lot. Opera allowed us to go in via the DevTools protocol.

Now, it meant that, especially in the early days of WebDriver, we would need to maintain N: M bindings where N is the language bindings and M is the browsers we support. This is not a road to a good product. We decided that we need something that every language would understand. We also needed something that would be pretty robust. HTTP was chosen and we set about building the JSONWireProtocol.

The JSONWireProtocol is where we built a REST-ish interface that would speak JSON. I say REST-ish because it didn't follow all the principles of REST but enough to make it powerful for our needs.

How does it relate to things now?

The Web, the internet, and the world has moved on. Why hasn't Selenium?

This is a good question and the thing is we are trying to move things on. Unfortunately the web has a state where it is broken unless it is working. HTTP is pretty robust as a protocol. It can also allow people to build up clusters for testing without having to worry too much about how the multiplexing would work. This is the reason why Selenium Grid was created and is still a pretty good choice when it comes to farming out your testing to multiple devices and multiple machines.

But is more like the web, be like them.

So... There are tools that use Chrome's Debug Protocol to drive the browser and some of the things they do better than Selenium are down to their choice of how they speak to the speak to the browser. Unfortunately it's a Chrome proprietary protocol and Google are not interested in working with other browsers on making it not.

Also, ignoring the interesting design choices from the the Google team, there is the problem that we have to have a permenantly open connection. In this case it uses WebSockets but if you remember my comment earlier about the internet is down until it's up. WebSockets would be constantly re-establishing the connection. There is also the problem of how much traffic would be going up and down that pipe.

This is fine for puppeteer where you are only speaking to something on your local machine but if you are combining a CI service, like Circle CI or TravisCI and something like AWS Device Farm, Sauce Labs, or BrowserStack you suddenly have a lot of internet inbetween you and your runner and that data needs to get somewhere.

The W3C Browser Testing and Tools Working group, which is made up of browser vendors and Selenium folk, are trying to design what this will look like to make sure that we can make it cross browser from the start without having to do weird hacky patches to browsers and ship those browsers ourselves.

Want to read more?

WebDriver Specification details about transport
Also other episodes are appearing on my blog before they on here

DEV Community: David Burns

Flakiness isn't from your test framework

Single threadedness of JavaScript

Transport layers

but what about CDP?

How do we solve flakiness then?

Front end frameworks hate testers

Asynchronocity is hard

Finally… work in the same world as your users

Engineering culture is hard to make but easy to destroy

Bullying is never ok

Building a healthy culture

Words matter

Leaders need to be at the front

Culture can be destroyed in an instant

Finally

I am on Twitch - Come work on Selenium with me!

Why?

What have I done so far?

When

Future Sessions

Browser Testing and Tools WG Meeting @ TPAC 2020

How Selenium Works: Episode 7 - Driver Executables

Why do I need to download something?

So... Why not just get the client bindings to speak whatever is native to the browser?

What about automatic downloads from the Selenium Project?

How Selenium Works: Episode 6 - sendKeys

You have an element you want to send some text, now what?

Check Element is still on the page.

Check the element is visible

Scroll to the element

Check if the element is interactable

Process Text String

Firing some events off

What about navigation?

Exceptions

File Inputs

IME Keyboards

Further Reading

How Selenium Works: Episode 5 - clicking

You have an element you want to click, now what?

Check Element is still on the page.

Check the element is visible

Scroll to the element

Check if the element is interactable

Firing some events off

What about navigation?

Further Reading

How Selenium Works: Episode 4 - Finding Elements

Finding an element is easy right?

Search Types

What happens when it finds the element?

and when it doesn't find the element or elements

Coming Soon: Relative locators

Further reading

How Selenium Works: Episode 3 - isDisplayed

Why do we care if it is "displayed"?

How does it work?

Why the recursion?

Can't we just ask the browser if an element is visible?

Further reading

How Selenium Works: Episode 2 - Navigation

driver.get

Certificates

Loading

Done...

What about JavaScript Frameworks and navigation

Conclusion

Further Reading

How Selenium Works: Episode 1 - Transportation

How Does Selenium talk to the browser?

HTTP and REST-ish? Really?

How does it relate to things now?

But is more like the web, be like them.

Want to read more?

`driver.get`