Selenium Chrome Proxy Authentication in Python
Setting chromedriver proxy with Selenium using Python
If you need to use a proxy with python and Selenium library with chromedriver you usually use the following code (Without any username and password:
| chrome_options = webdriver.ChromeOptions() | |
| chrome_options.add_argument('--proxy-server=%s' % hostname + ":" + port) | |
| driver = webdriver.Chrome(chrome_options=chrome_options) |
This works fine unless proxy requires authentication. if the proxy requires you to log in with a username and password it will not work. In this case, you have to use more tricky solution that is explained below. By the way, if you whitelist your server IP address from the proxy provider or server it should not ask proxy credentials.
HTTP Proxy Authentication with Chromedriver in Selenium
To set up proxy authentication we will generate a special file and upload it to chromedriver dynamically using the following code below. This code configures selenium with chromedriver to use HTTP proxy that requires authentication with user/password pair.
| import os | |
| import zipfile | |
| from selenium import webdriver | |
| PROXY_HOST = '192.168.3.2' # rotating proxy or host | |
| PROXY_PORT = 8080 # port | |
| PROXY_USER = 'proxy-user' # username | |
| PROXY_PASS = 'proxy-password' # password | |
| manifest_json = """ | |
| { | |
| "version": "1.0.0", | |
| "manifest_version": 2, | |
| "name": "Chrome Proxy", | |
| "permissions": [ | |
| "proxy", | |
| "tabs", | |
| "unlimitedStorage", | |
| "storage", | |
| "<all_urls>", | |
| "webRequest", | |
| "webRequestBlocking" | |
| ], | |
| "background": { | |
| "scripts": ["background.js"] | |
| }, | |
| "minimum_chrome_version":"22.0.0" | |
| } | |
| """ | |
| background_js = """ | |
| var config = { | |
| mode: "fixed_servers", | |
| rules: { | |
| singleProxy: { | |
| scheme: "http", | |
| host: "%s", | |
| port: parseInt(%s) | |
| }, | |
| bypassList: ["localhost"] | |
| } | |
| }; | |
| chrome.proxy.settings.set({value: config, scope: "regular"}, function() {}); | |
| function callbackFn(details) { | |
| return { | |
| authCredentials: { | |
| username: "%s", | |
| password: "%s" | |
| } | |
| }; | |
| } | |
| chrome.webRequest.onAuthRequired.addListener( | |
| callbackFn, | |
| {urls: ["<all_urls>"]}, | |
| ['blocking'] | |
| ); | |
| """ % (PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS) | |
| def get_chromedriver(use_proxy=False, user_agent=None): | |
| path = os.path.dirname(os.path.abspath(__file__)) | |
| chrome_options = webdriver.ChromeOptions() | |
| if use_proxy: | |
| pluginfile = 'proxy_auth_plugin.zip' | |
| with zipfile.ZipFile(pluginfile, 'w') as zp: | |
| zp.writestr("manifest.json", manifest_json) | |
| zp.writestr("background.js", background_js) | |
| chrome_options.add_extension(pluginfile) | |
| if user_agent: | |
| chrome_options.add_argument('--user-agent=%s' % user_agent) | |
| driver = webdriver.Chrome( | |
| os.path.join(path, 'chromedriver'), | |
| chrome_options=chrome_options) | |
| return driver | |
| def main(): | |
| driver = get_chromedriver(use_proxy=True) | |
| #driver.get('https://www.google.com/search?q=my+ip+address') | |
| driver.get('https://httpbin.org/ip') | |
| if __name__ == '__main__': | |
| main() |
Function get_chromedriver returns configured selenium webdriver that you can use in your application. This code is tested and works just fine.
Read more about onAuthRequired event in Chrome.
Top comments (1)
How to use with options --headless ?
If i add chrome_options.add_argument('--headless')
I see in logs, when create driver:
{WebDriverException}Message: unknown error: failed to wait for extension background page to load: chrome-extension://bebklmomgilochanmemgcdlodnpmlpho/_generated_background_page.html