Discussion on: Creating an API that runs Selenium via AWS Lambda

[Jaira Encio]

does the error only occur when you add selenium-stealth library? Upon checking I noticed that others are experiencing issue in their local machines just by using stealth. You could try adding options.add_argument("--disable-blink-features=AutomationControlled") . Then try if it works both on your local and lambda.

[Awolad Hossain]

Yes.

With the selenium-stealth default options like following:

options = Options()
options.binary_location = '/opt/headless-chromium'
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
# options.add_argument("--headless")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)

driver = webdriver.Chrome('/opt/chromedriver', chrome_options=options)

stealth(driver,
        languages=["en-US", "en"],
        vendor="Google Inc.",
        platform="Win32",
        webgl_vendor="Intel Inc.",
        renderer="Intel Iris OpenGL Engine",
        fix_hairline=True,
        )

I'm getting error: Message: unknown error: Chrome failed to start: exited abnormally\n (Driver info: .....

By using this post options like following:

options = Options()
options.binary_location = '/opt/headless-chromium'
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--single-process')
options.add_argument('--disable-dev-shm-usage')

driver = webdriver.Chrome('/opt/chromedriver', chrome_options=options)

stealth(driver,
     languages=["en-US", "en"],
     vendor="Google Inc.",
      platform="Win32",
      webgl_vendor="Intel Inc.",
      renderer="Intel Iris OpenGL Engine",
       fix_hairline=True,
)

I'm getting error: "'WebDriver' object has no attribute 'execute_cdp_cmd'"

[Jaira Encio]

I'm seeing this article related to "execute_cdp_cmd" error. Apparently they used pip install --pre selenium to be able to execute CDP commands github.com/SeleniumHQ/selenium/iss...

[Awolad Hossain]

I also tried that but not working. I forgot to mention that. It would be helpful for us if you try with the selenium-stealth package and update this post. Because some websites we can't scrape without the selenium-stealth package. Thanks!