DEV Community

Jaira Encio for AWS Community ASEAN

Posted on • Updated on

Creating an API that runs Selenium via AWS Lambda

Being an automation tester, my job is to automate everything. As I was running my test script via terminal I realised that I’m the only who can execute the scripts I made. What if someone wants to run it? like the Devs, Project Manager, etc. It would be a tedious task to clone my repo, install libraries, and run the script. So I decided that maybe I can store my test script inside a serverless machine and make it accessible via API request.

I experimented with various AWS resources such as creating my own lambda function, checking features of API gateway, Codepipeline, etc. After several attempts, I was finally able to run my script inside. And then I just researched how to access my lambda via API.

This will result in higher production and time savings. Engineers may focus on vital work because automated testing does not require human interaction. This is like a portable testing device that anyone could execute. With fast test execution, developers get testing reports in an instant, so whenever a failure occurs, they will react to it quickly. Test automation will make it easier to update programs quickly. As a result, automated testing leads to increased team responsiveness, improved user experience, and increased customer satisfaction.

Overview

  1. Create 2 Lambda layers that has selenium and chromedriver libraries
  2. Include created lambda layers in serverless.yml of lambda then deploy

Creating Selenium Lambda Layer

Place libraries in python/lib/python3.6/site-packages/ to include them in a layer.

Download Selenium to layer directory

pip3.6 install -t selenium/python/lib/python3.6/site-packages selenium==3.8.0
cd selenium
zip -r python.zip python/
Enter fullscreen mode Exit fullscreen mode

Once finished, Create lambda layer then upload zip file

1. Go to AWS Console Lambda/Layers
2. Click Create Layer
3. Input the following in the layer configuration

Name: selenium
Description: Selenium layer
Upload zip file created: python.zip
Compatible runtimes: Python 3.6
Enter fullscreen mode Exit fullscreen mode

selenium-layer

4. Click Create

Note: You can user whatever version you prefer, you just need to select compatible runtime when uploading your package

 Creating Chromedriver Lambda layer

Download chrome driver

mkdir -p chromedriver
cd chromedriver
curl -SL https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip > chromedriver.zip
unzip chromedriver.zip
rm chromedriver.zip
Enter fullscreen mode Exit fullscreen mode

Download chrome binary

curl -SL https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-41/stable-headless-chromium-amazonlinux-2017-03.zip > headless-chromium.zip
unzip headless-chromium.zip
rm headless-chromium.zip
Enter fullscreen mode Exit fullscreen mode

Compress driver and binary

ls
chromedriver headless-chromium
zip -r chromedriver.zip chromedriver headless-chromium
Enter fullscreen mode Exit fullscreen mode

Once finished, Create lambda layer then upload zip file

1. Go to AWS Console Lambda/Layers
2. Click Create Layer
3. Input the following in the layer configuration

Name: chromedriver
Description: chrome driver and binary layer
Upload zip file created: chromedriver.zip
Compatible runtimes: Python 3.6
Enter fullscreen mode Exit fullscreen mode

chrome-layer

4. Click Create

 Creating Lambda Function

To ensure that your function code has access to libraries included in layers, Lambda runtimes include paths in the '/opt' directory.

 File Structure

── /lambda/            # lambda function
  ├── /handler.py      # source code of lambda function 
  └── /serverless.yaml # serverless config
Enter fullscreen mode Exit fullscreen mode

 Code

Copy the code below to /lambda/handler.py

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

def main(event, context):
    options = Options()
    options.binary_location = '/opt/headless-chromium'
    options.add_argument('--headless')
    options.add_argument('--no-sandbox')
    options.add_argument('--single-process')
    options.add_argument('--disable-dev-shm-usage')

    driver = webdriver.Chrome('/opt/chromedriver',chrome_options=options)
    driver.get('https://www.google.com/')

    driver.close();
    driver.quit();

    response = {
        "statusCode": 200,
        "body": "Selenium Headless Chrome Initialized"
    }

    return response
Enter fullscreen mode Exit fullscreen mode

Copy the code below to /lambda/serverless.yaml.

service: selenium-lambda

provider:
  name: aws
  runtime: python3.6
  region: ap-southeast-2
  timeout: 900

functions:
  main:
    memorySize: 1000
    handler: handler.main
    events:
      - http:
          path: test
          method: get

    layers:
      - arn:aws:lambda:ap-southeast-2:{}:layer:chromedriver:2
      - arn:aws:lambda:ap-southeast-2:{}:layer:selenium:2

resources:
  Resources:
    ApiGatewayRestApi:
      Properties:
        BinaryMediaTypes:
          - "*/*"
Enter fullscreen mode Exit fullscreen mode

 Deploy Lambda Function

Go to /lambda directory

$ sls deploy
Enter fullscreen mode Exit fullscreen mode

 Output

Serverless: Stack update finished...
Service Information
service: selenium-lambda
stage: dev
region: ap-southeast-2
stack: selenium-lambda-dev
api keys:
  None
endpoints:
  GET - https://{name}.execute-api.ap-southeast-2.amazonaws.com/dev/test
functions:
  main: selenium-lambda-dev-main
Enter fullscreen mode Exit fullscreen mode

output

You should get same response as below when accessing API

{
"statusCode": 200,
"body": "Selenium Headless Chrome Initialized"
}
Enter fullscreen mode Exit fullscreen mode

This deployment automatically creates cloudformation stack and s3 bucket.

Cloudformation stack

S3

Linkedin
Email: jairaencio@gmail.com

Discussion (29)

Collapse
awolad profile image
Awolad Hossain

@jairaencio It's working great. But I can't use the selenium-stealth plugin. Getting an error. Message: unknown error: Chrome failed to start: exited abnormally\n (Driver info: .....

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium_stealth import stealth

def lambda_handler(event, context):
    options = Options()
    options.binary_location = '/opt/headless-chromium'    
    options.add_argument("start-maximized")
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.add_experimental_option('useAutomationExtension', False)

    driver = webdriver.Chrome('/opt/chromedriver',chrome_options=options)

    stealth(driver,
        languages=["en-US", "en"],
        vendor="Google Inc.",
        platform="Win32",
        webgl_vendor="Intel Inc.",
        renderer="Intel Iris OpenGL Engine",
        fix_hairline=True,
        )

    driver.get('https://quizlet.com/446134722/it-management-flash-cards/')

    driver.close();
    driver.quit();

    response = {
        "statusCode": 200,
        "body": "Selenium Headless Chrome Initialized"
    }

    return response
Enter fullscreen mode Exit fullscreen mode
Collapse
jairaencio profile image
Jaira Encio Author

Hi @awolad I think you need to include the stealth library package in your lambda layer. Notice in my tutorial I have 2 different lambda layers for my selenium and chromedriver package. You can create another lambda layer or just simply include it in the 2 layers

Collapse
awolad profile image
Awolad Hossain

@jairaencio Yes, I've added the stealth library package in the selenium lambda layer. There is no import error.

Thread Thread
jairaencio profile image
Jaira Encio Author

Great! Always happy to help :)

Thread Thread
awolad profile image
Awolad Hossain

@jairaencio Sorry, It's not solved yet. I mean the error is not related to the import the stealth package issue. Because the package is already in my lambda layer. The driver fails to load when the stealth package is used.

Thread Thread
jairaencio profile image
Jaira Encio Author • Edited on

does the error only occur when you add selenium-stealth library? Upon checking I noticed that others are experiencing issue in their local machines just by using stealth. You could try adding options.add_argument("--disable-blink-features=AutomationControlled") . Then try if it works both on your local and lambda.

Thread Thread
awolad profile image
Awolad Hossain

Yes.

With the selenium-stealth default options like following:

options = Options()
options.binary_location = '/opt/headless-chromium'
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
# options.add_argument("--headless")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)

driver = webdriver.Chrome('/opt/chromedriver', chrome_options=options)

stealth(driver,
        languages=["en-US", "en"],
        vendor="Google Inc.",
        platform="Win32",
        webgl_vendor="Intel Inc.",
        renderer="Intel Iris OpenGL Engine",
        fix_hairline=True,
        )
Enter fullscreen mode Exit fullscreen mode

I'm getting error: Message: unknown error: Chrome failed to start: exited abnormally\n (Driver info: .....

By using this post options like following:

options = Options()
options.binary_location = '/opt/headless-chromium'
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--single-process')
options.add_argument('--disable-dev-shm-usage')

driver = webdriver.Chrome('/opt/chromedriver', chrome_options=options)

stealth(driver,
     languages=["en-US", "en"],
     vendor="Google Inc.",
      platform="Win32",
      webgl_vendor="Intel Inc.",
      renderer="Intel Iris OpenGL Engine",
       fix_hairline=True,
)
Enter fullscreen mode Exit fullscreen mode

I'm getting error: "'WebDriver' object has no attribute 'execute_cdp_cmd'"

Thread Thread
jairaencio profile image
Jaira Encio Author

I'm seeing this article related to "execute_cdp_cmd" error. Apparently they used pip install --pre selenium to be able to execute CDP commands github.com/SeleniumHQ/selenium/iss...

Thread Thread
awolad profile image
Awolad Hossain

I also tried that but not working. I forgot to mention that. It would be helpful for us if you try with the selenium-stealth package and update this post. Because some websites we can't scrape without the selenium-stealth package. Thanks!

Collapse
prenitwankhede profile image
prenit-wankhede

Thanks a ton brother for simple yet elegant walk through.
I have been trying so many tutorials and ways to get it to work but no luck.

With selenium version, chromedriver version and headless-chrome version as mentioned in the post, finally got it working. Thanks a bunch !

Collapse
da_shen_a7cf582bace0b4404 profile image
Da Shen

tested working.. good article. note that Python runtime has to be 3.6. It won't work otherwise.

Collapse
quibski profile image
quibski

Surely many Devs and QA will benefit from this. Hopefully a demo can be made/shown

Collapse
tchua profile image
tchua

+1 to this, a demo would be great!

Collapse
jairaencio profile image
Jaira Encio Author

uhm hahaha

Collapse
rolinj profile image
rolinj

Kudos Jai! Great tutorial indeed!
For reference, may you add as well some screenshots of the created cloudformation stack and s3 bucket on the output section? Thanks :D

Collapse
jairaencio profile image
Jaira Encio Author

Uploaded screenshots of cloudformation and s3 bucket. Thanks for the feedback :)

Collapse
ivy07 profile image
Ivy ☕

Great help! Will definitely need more articles like this un the future.

Collapse
raphael_jambalos profile image
Raphael Jambalos

The article is very helpful! It brings automation to the next level. By having running automated tests in a more automated way, developers will be empowered to make sure their code runs optimally.

Collapse
jlgarcia profile image
jltuts

Good job jai! Very helpful!

Collapse
lunchcodes profile image
LunchCodes

Exactly what I needed! Thank you!

Collapse
youngcto profile image
youngcto

Live Demo here: youtu.be/qIcVGDEjtt4?t=3482 (part of the June Meetup)

Collapse
chrisjeriel profile image
chrisjeriel

Great work! A well-thought-out article, straightforward and concise. Looking forward more advanced implementations.

Collapse
em__sia profile image
e

Naks! Great job! Screenshots will be helpful too. And a milk-tea will do. Thanks! Ahahahhahaha. :)

Collapse
chris93007wq profile image
Christine John

I'm getting the following error -
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot find Chrome binary
(Driver info: chromedriver=2.37.544315(730aa6a5fdba159ac9f4c1e8cbc59bf1b5ce12b7),platform=Linux 4.14.255-276-224.499.amzn2.x86_64 x86_64)

Could someone help me please? :(

Collapse
chris93007wq profile image
Christine John
Collapse
ranzeyxc profile image
ranzey

Huge help! :)

Collapse
jobad profile image
Badjo Badiola

This is AWSome! thank you!

Collapse
silencer017 profile image
silencer017

Well done, this is very informative!

Collapse
chiggy_wiggy profile image
Maynard Prepotente • Edited on

Great Job! Will definitely help a lot of Devs and QA! Adding screenshots of the output will make the job easier tho =)