Discussion on: Creating an API that runs Selenium via AWS Lambda

[Achim Grolimund]

Hey @jairaencio
Hello everyone, thanks for the guide. but it seems to me that this method no longer works without a Dockerimage.

I keep getting the error:

{
  "errorMessage": "Message: Service /opt/chromedriver unexpectedly exited. Status code was: 127\n",
  "errorType": "WebDriverException",
  "requestId": "ef0d3b0d-ee3f-4822-ba11-9dd40920680b",
  "stackTrace": [
    "  File \"/var/task/app.py\", line 29, in main\n    driver = webdriver.Chrome(service=s, options=op)\n",
    "  File \"/opt/python/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py\", line 69, in __init__\n    super().__init__(DesiredCapabilities.CHROME['browserName'], \"goog\",\n",
    "  File \"/opt/python/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py\", line 89, in __init__\n    self.service.start()\n",
    "  File \"/opt/python/lib/python3.9/site-packages/selenium/webdriver/common/service.py\", line 98, in start\n    self.assert_process_still_running()\n",
    "  File \"/opt/python/lib/python3.9/site-packages/selenium/webdriver/common/service.py\", line 110, in assert_process_still_running\n    raise WebDriverException(\n"
  ]
}

Python 3.9
Selenium 4.5.0
chromedriver 106.0.5249.61
headless-chromium v1.0.0-57

This is My Makefile to create all the stuff i need. At the end i do upload all what is inside the lambda Folder (Code + 2 Layers)

BOLD := \033[1m
NORMAL := \033[0m
GREEN := \033[1;32m

.DEFAULT_GOAL := help
HELP_TARGET_DEPTH ?= \#
.PHONY: help
help: # Show how to get started & what targets are available
    @printf "This is a list of all the make targets that you can run, e.g. $(BOLD)make dagger$(NORMAL) - or $(BOLD)m dagger$(NORMAL)\n\n"
    @awk -F':+ |$(HELP_TARGET_DEPTH)' '/^[0-9a-zA-Z._%-]+:+.+$(HELP_TARGET_DEPTH).+$$/ { printf "$(GREEN)%-20s\033[0m %s\n", $$1, $$3 }' $(MAKEFILE_LIST) | sort
    @echo


install: clean # Format all files (terraform fmt --recursive .)
    python3.9 -m pip install -t python/lib/python3.9/site-packages selenium==4.5.0 --upgrade
    python3.9 -m pip install -t python/lib/python3.9/site-packages wget --upgrade
    curl -SL https://chromedriver.storage.googleapis.com/106.0.5249.61/chromedriver_linux64.zip > chromedriver.zip
    curl -SL https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-57/stable-headless-chromium-amazonlinux-2.zip > headless-chromium.zip
    mkdir -p lambda
    zip -9 -r lambda/python.zip python
    unzip chromedriver.zip
    unzip headless-chromium.zip
    zip -9 -r lambda/chromedriver.zip chromedriver headless-chromium
    rm -f chromedriver.zip headless-chromium.zip chromedriver headless-chromium

build: # Create all layers to upload to the Lambda Function
    zip -9 -r lambda/app.zip app.py

clean:
    rm -rf lambda python

And here is a overview of my python script:

import re
import wget

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait


def main(event, context):
    regex = r"[ \w-]+?(?=\.txt\.gz)"

    s = Service('/opt/chromedriver')
    op = webdriver.ChromeOptions()
    op.binary_location = '/opt/headless-chromium'
    op.add_argument('--headless')
    op.add_argument('--no-sandbox')
    op.add_argument('--disable-dev-shm-usage')
    op.add_argument('--disable-gpu')
    op.add_argument('--disable-dev-tools')
    op.add_argument("--disable-extensions")
    op.add_argument('--no-zygote')
    op.add_argument('--single-process')
    op.add_argument('--enable-logging')
    op.add_argument('--log-level=0')
    op.add_argument("--disable-notifications")
    op.add_argument('--v=99')
    driver = webdriver.Chrome(service=s, options=op)

    driver.get(
        "https://xxxxxx")

    (.....)

    driver.close()

Is there another way to run it directly in a lambda without a docker image?

Best Regards

[Rocco]

I have the configuration propose on the edited version of the article but I get the same error. I created all the version on an amazon linux using the python3.9 command like in the example but I keep getting the same error.

Did you find any solution?

[Achim Grolimund]

Yes it needs an special tool inside the docker image. I wil post it here today as soon as im on my computer

[Nadia-Ou]

did you find any solution, please?

[Jaira Encio]

Great take on this achin💪 I still havent gotten back on this but your method of using dockerimage is very useful for everyone as well 👏🏽

[rajans163]

Hi Dear...were you able to get resolution of this issue. I am getting the same issue in my lambda.
Please share the resolution as I am completely stuck.