DEV Community: Marcin K.

Mysterious "chrome not reachable" WebDriverError in Selenium

Marcin K. — Sun, 12 Apr 2020 13:27:00 +0000

Today I would like to share what I have learned regarding "chrome not reachable" error.

I came across this one when I tried to find an element right after navigating to a particular address. Stacktrace looked like this:

Traceback (most recent call last):
        ...
         9: from /Users/marcin/.gem/ruby/2.6.5/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/common/search_context.rb:62:in `find_element'
         8: from /Users/marcin/.gem/ruby/2.6.5/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/oss/bridge.rb:554:in `find_element_by'
         7: from /Users/marcin/.gem/ruby/2.6.5/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/oss/bridge.rb:587:in `execute'
         6: from /Users/marcin/.gem/ruby/2.6.5/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/bridge.rb:167:in `execute'
         5: from /Users/marcin/.gem/ruby/2.6.5/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/common.rb:64:in `call'
         4: from /Users/marcin/.gem/ruby/2.6.5/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/default.rb:114:in `request'
         3: from /Users/marcin/.gem/ruby/2.6.5/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/common.rb:88:in `create_response'
         2: from /Users/marcin/.gem/ruby/2.6.5/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/http/common.rb:88:in `new'
         1: from /Users/marcin/.gem/ruby/2.6.5/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/response.rb:34:in `initialize'
/Users/marcin/.gem/ruby/2.6.5/gems/selenium-webdriver-3.142.7/lib/selenium/webdriver/remote/response.rb:72:in `assert_ok': chrome not reachable (Selenium::WebDriver::Error::WebDriverError)
  (Session info: chrome=81.0.4044.92)
  (Driver info: chromedriver=2.37.544337 (8c0344a12e552148c185f7d5117db1f28d6c9e85),platform=Mac OS X 10.15.2 x86_64)

As you can see from the above, my stack is:

Chrome 81.0.4044.92
Chromedriver 2.37.544337
MacOS 10.15.2 x86_64
Ruby 2.6.5
Selenium Webdriver gem 3.142.7

I can't provide the exact site on which I was able to replicate it because it appeared only after logging in. However, the code looked similar to this:

Solution 1:

Remove --single-process option.
Enabling this option means that the renderer and plugins will run in the same process as the browser. This may be useful if you're for instance running your code in a serverless function but in that case, you will probably not use standard chrome anyway.

Solution 2:

Use Chromium binary instead of Chrome.
Chromium is the open-source project, that the Chrome browser is based on. Google added more features to its' browser, as more media support, plugins, etc. You can read more here.
I used chromium version 84.0.4113.0 (Developer Build) (64-bit).
An interesting thing is that --single-process flag worked for Chromium, so it looks like Chrome renderer had more work to do.

Though I have not found the precise cause of this error, both of these solutions worked for me. I hope you will find it useful.

Things I have learned working on a project for my younger brother

Marcin K. — Sun, 01 Dec 2019 11:52:00 +0000

It all started about a year ago. I have met with my younger brother Paweł. He had this idea "that would make us rich" and asked if I could implement it. He is really into sports betting, he claimed he has a system for finding when the underdog team is about to score the next goal. The concept was straight forward - the app would take the data from live football events and analyze them. If the stats are right, it sends an email.
Looking at the data he had gathered, I knew that it was not enough to tell how accurate this system is, but I thought "that's a great learning opportunity!" so I wanted to give it a try anyway.
That's how I began working on a non-work related project "requested" by my brother. (If you thought based on the title that I would be doing something that my brother was supposed to do for his school/studies, I am sorry to disappoint you)

I wanted to share some lessons I have learned on the way. The list is very subjective and most of the stuff may sound obvious (but we do not always go with obvious things, do we?). Still, I hope some of you will find it beneficial:

Requirements WILL change - "what do you mean it's not working right? I have implemented it the way you have described it the docs you have shared" "right... but it was supposed to take the stats only from the second half". So what do you do? I really liked the recommendations given by Uncle Bob in his book "Clean architecture" - divide the app into modules based on their abstraction level and note the flow of control. Our app was scraping the stats from an external betting website. When they changed the design, I needed to change only one class with scraping logic to make it work again.
Follow good practices very early - by using linter and taking care of high test coverage from the start you save yourself some work.
Try out a new technology - that was my first experience with Bitbucket pipelines, AWS EC2 and AWS CodeDeploy, as well as Selenium for web scraping. I feel like I have learned a lot.
Track your time - I used Toggl for measuring the time I have spent on this project. I feel like it helped me focus on the task, but also be a little better at planning and estimating my work.
When you're scraping - don't get banned! - You probably know that your app should not send many HTTP requests in a short time. I knew that from the start, but then the instance with our app would get banned. Why? The app tried to refresh the external website when the elements are missing. Imagine, what had happened if that website was under heavy load! So be aware of it on every level of your app (Captain obvious strikes again!). There is more to web scraping than just the interval of sending requests: I recommend the artice if you want to learn more.
Track your budget - it refers to cloud services. Can you use a free trial for your project? When is it going to end? How much a month would cost me? You can also use a pricing calculator when you're not sure. I highly recommend setting up limits/alerts on your account. Oh, and you can always create a new Google Cloud or AWS account when you want to try something new in the free trial.

Our app turned out to be pretty inaccurate in predicting the next team to score (Tossing a coin would give us similar results...), so we have decided to terminate the instance recently. Even though the app did not make us rich, I feel satisfied that I took my time and I was there through the whole lifecycle of this "product".
Since the app is unlikely to be monetized now, I put the code on Github. Feel free to play around if you would like to.

Thanks for reading!

Serverless web scraper in Ruby - tutorial

Marcin K. — Sat, 05 Oct 2019 16:06:37 +0000

Imagine you have this awesome web app that will make you very rich someday. This app has some end-user tests. You used Selenium to automate all the manual stuff requiring a browser interaction.
As your app gets bigger, end-user tests are taking more and more time that could be spent on something else.
You recruit more QA engineers, and they all have to configure selenium, chrome driver, and a proper browser binary. This gets cumbersome and error-prone.

Why not run those tests in parallel? Why not keep them totally separate from our app? Why not keep them, configure them and run them on a separate machine?

We can do it with serverless chrome!
It's just a chrome binary designed to be used on AWS Lambda (at the moment of writing this article GCP and Microsoft Azure is not yet supported).
Let's build a very simple web scraping app with it. We are going to write it in Ruby - If you prefer writing in Python, here's an article for you.

Requirements:

AWS account (and some very basic knowledge)
Ruby installed (version 2.5.x)
Serverless chrome (1.0.0-37)
Chromedriver (2.37)
Ruby gems: selenium-webdriver (I used 3.142.4) and bundler (2.0.x)

Note: Your lambda function, s3 bucket, role, and the user should be created in the same region.

Create role

Create an IAM role and attach to it existing AWSLambdaFullAccess policy. Here is a tutorial for creating roles.

Create a user with programmatic access

Create an AWS user and attach to it the same policy as above.
Here is a tutorial for adding new users.
Write the access id and secret that you have obtained. Set them as the environment variables (link) and configure them in your AWS profiles.

Create an S3 bucket

Create an AWS S3 bucket. Here is how to do it.

Create a lambda function

Now let's go to the AWS console again and create our lambda function.

When asked to enter the basic information for your function, add a name you want and choose Ruby 2.5.

Once it's created, go to the "Basic settings" in the function view and set the memory to 512MB and the timeout to 1 min.

Have a look at the template for our function:

Also, assign the role that we have created earlier as the execution role.

Install chromedriver and serverless chrome

Let's grab serverless chrome:

wget https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-37/stable-headless-chromium-amazonlinux-2017-03.zip
unzip stable-headless-chromium-amazonlinux-2017-03.zip -d bin/
rm stable-headless-chromium-amazonlinux-2017-03.zip

And the chromedriver:

wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
unzip chromedriver_linux64.zip -d bin/
rm chromedriver_linux64.zip

Install selenium web driver gem

First, we need to create a Gemfile for our project with the following content:

Once we have the Gemfile, we can install the required gem and its dependencies:

bundle install --path vendor/bundle

Note the location we are installing it to. We will need to include this folder in the package deployed to AWS S3.

Implement the scraper

The first thing we will need is to set up a selenium driver.
Note how we are passing paths to the binaries we have just installed.

Next, let's implement the lambda function itself.

Here I am just using selenium webdriver API to send some input into google.com and show me the browser title. If you would like to know the details or experiment a little with it, check out the selenium API docs. Do not forget to tell the driver to quit in the end!

I am just going to add some additional driver options to make it more efficient:

Run the scraper code locally

If you're using macOS or Windows, you will need to test your code with Docker. The good news is that there is an image provided by AWS that mirrors the lambda environment and we can use it directly. We use --mount flag here to set /dev/shm permissions to read-only.

docker run --rm -v "$PWD":/var/task --mount type=tmpfs,target=/dev/shm,readonly=true lambci/lambda:ruby2.5 lambda_function.lambda_handler

Upload to lambda

Run those commands in order to zip our code and its dependencies, upload it to S3, and update our function from there.

We are ready to invoke our function! Do it with aws lambda invoke --function-name your_function_name output_file in your terminal, or use the "Test" button in the function view in the AWS console.

Happy scraping!

Deploying Ruby app using AWS Codedeploy to EC2: How to do it with Bundler!

Marcin K. — Wed, 10 Jul 2019 16:43:20 +0000

It's late at night, after a week of struggle I managed to configure my deployment properly. The purpose of this short article is to show you some problems that I came across with dependencies in my Ruby app during deployment with Codedeploy, and solutions to them.

Prerequisites: Basic knowledge of AWS Codedeploy configuration (If you went through the documentation you will do just fine).

From the beginning...

I have my Ruby app. It is using Bundler to install gem dependencies. I put it on AWS EC2 and it works just great.
Note: On my EC2 I am using Amazon Linux AMI 2018.03.0 (HVM), SSD Volume Type, since it comes with Ruby and some CLI tools installed out of the box.
There is just one thing missing to make me happy - I would like to deploy my application when a particular branch is updated on my bitbucket repository.
Let's assume I have VM running and codedeploy-agent installed on it.
I recommend this tutorial to set up deployment pipelines for your bitbucket repository.
There is one thing not covered in the tutorial though: installing app dependencies with Bundler.

The problem #1

So what's the big deal?
I can use gem install bundler and then add bundle install in my scripts ran in my AppSpec deployment hooks, right?

(App code is not really important here)
But I get an error when I deploy my code:

[2019-07-07 10:01:40.378] [d-0N2XZ4E6A]LifecycleEvent - AfterInstall
[2019-07-07 10:01:40.379] [d-0N2XZ4E6A]Script - scripts/bundle_install
[2019-07-07 10:01:40.391] [d-0N2XZ4E6A][stderr]/opt/codedeploy-agent/deployment-root/d0556b49-b646-4579-9d8e-20fca9397a2e/d-0N2XZ4E6A/deployment-archive/scripts/bundle_install: line 4: bundle: command not found
[2019-07-07 10:03:58.992] [d-8Y4VW7E6A]LifecycleEvent - ApplicationStop
[2019-07-07 10:03:58.992] [d-8Y4VW7E6A]Script - scripts/stop_server

It looks like bundle binary is not found.

Solution

So let's just install the binary somewhere in the environment PATH:

Problem #2

All right, so will it work now?

2019-07-07 18:00:20 [stdout]Fetching json 2.2.0
2019-07-07 18:00:20 [stdout]Installing json 2.2.0 with native extensions
2019-07-07 18:00:20 [stderr]Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
2019-07-07 18:00:20 [stderr]
2019-07-07 18:00:20 [stderr]current directory:
2019-07-07 18:00:20 [stderr]/home/ec2-user/vendor/bundle/ruby/2.4/gems/json-2.2.0/ext/json/ext/generator
2019-07-07 18:00:20 [stderr]/usr/bin/ruby2.4 -r ./siteconf20190707-26032-sw7ot0.rb extconf.rb
2019-07-07 18:00:20 [stderr]mkmf.rb can't find header files for ruby at /usr/share/ruby/include/ruby.h
2019-07-07 18:00:20 [stderr]
2019-07-07 18:00:20 [stderr]extconf failed, exit code 1
2019-07-07 18:00:20 [stderr]
2019-07-07 18:00:20 [stderr]Gem files will remain installed in
2019-07-07 18:00:20 [stderr]/home/ec2-user/vendor/bundle/ruby/2.4/gems/json-2.2.0 for inspection.
2019-07-07 18:00:20 [stderr]Results logged to
2019-07-07 18:00:20 [stderr]/home/ec2-user/vendor/bundle/ruby/2.4/extensions/x86_64-linux/2.4/json-2.2.0/gem_make.out
2019-07-07 18:00:20 [stderr]
2019-07-07 18:00:20 [stderr]An error occurred while installing json (2.2.0), and Bundler cannot continue.
2019-07-07 18:00:20 [stderr]Make sure that `gem install json -v '2.2.0' --source 'https://rubygems.org/'`
2019-07-07 18:00:20 [stderr]succeeds before bundling.
2019-07-07 18:00:20 [stderr]
2019-07-07 18:00:20 [stderr]In Gemfile:
2019-07-07 18:00:20 [stderr]  json

Not really. Funny thing is that some gems got installed, while some did not - in my case it's json gem that cannot be build.

Solution

I needed to install another two things to make it work - ruby development kit and c++ garbage collector

Why putting it all in hooks scripts?

Good question. Why not just connect to EC2 instance and install all this stuff by hand?
Because it will not work. I have not found a precise cause, but I have found out that the environment under which deployment is made (at least before copying/replacing files of the working app) is different than the one we have access to when connecting to our VM.

I hope you found the above information useful. If you did, please like, share or leave a comment.

That's my first post, so any constructive feedback will be appreciated :)