DEV Community

Leonardo Teixeira Menezes
Leonardo Teixeira Menezes

Posted on

Automated Dapps Scrapping with Selenium and Metamask

One of the newest trends in web development is the rise of decentralized applications, also known as Dapps. These applications are built leveraging decentralized networks in order to provide trustless interactions between users, using predefined interactions built as smart contracts (if you want to know more about dapps head over here).

To access Dapps users need to use a crypto wallet to connect with, this creates a new challenge for developers who want to scrap and/or test Dapps using tools like Selenium. In this post we will cover the basics on how to solve this using Python and Chromium, however, the principles described here can be applied using any programming language and web browser automation tool.

Most current Dapps depend on a crypto wallet being present in the user browser as an extension, injecting in the web page information about the user wallet and the network it's connected with. One of the most popular browser crypto wallet is Metamask. In order to successfully scrape a Dapp we need to interact not only with the target website but also with the Metamask extension simultaneously, to approve the app connection with our wallet and other possible transactions.

Compressing the extension

In order to load the extension on our automated browser we will first need to compress the Metamask extension to a .crx file, here are the steps:

  • Install Metamask on your regular chrome
  • Navigate to chrome://extensions/
  • Click 'Pack extension' and enter the local path to the Metamask extension This will generate a .crx file that you can use to load as an extension on Chromium. Save the name of the folder where the extension is installed, this will be the 'Extension ID' that we will use later.

Loading the extension

To load Chromium with Metamask installed run:

from selenium import webdriver

EXTENSION_PATH = 'ENTER THE PATH TO YOUR CRX FILE'
opt = webdriver.ChromeOptions()
opt.add_extension(EXTENSION_PATH)

driver = webdriver.Chrome(chrome_options=opt)
Enter fullscreen mode Exit fullscreen mode

Interacting with Metamask

In order to interact with the dapp and Metamask simultaneously, we will need to have multiple tabs in our Chromium, one for the target Dapp and another one for Metamask itself.

When Chromium starts it will have a welcome screen for the Metamask extension, which will prompt you to setup your wallet, this is an example code to import an existing wallet (You may need to update some steps depending on your Metamask version):

driver.find_element_by_xpath('//button[text()="Get Started"]').click()
driver.find_element_by_xpath('//button[text()="Import wallet"]').click()
driver.find_element_by_xpath('//button[text()="No Thanks"]').click()

# After this you will need to enter you wallet details

inputs = driver.find_elements_by_xpath('//input')
inputs[0].send_keys(SECRET_RECOVERY_PHRASE)
inputs[1].send_keys(NEW_PASSWORD)
inputs[2].send_keys(NEW_PASSWORD)
driver.find_element_by_css_selector('.first-time-flow__terms').click()
driver.find_element_by_xpath('//button[text()="Import"]').click()
driver.find_element_by_xpath('//button[text()="All Done"]').click()
Enter fullscreen mode Exit fullscreen mode

After this Metamask will be successfully set up in Chromium, ready to connect to a Dapp.
When you need to interact with Metamask again you will need to use it in a different tab, like so:

EXTENSION_ID = 'ENTER HERE THE EXTENSION ID THAT YOU SAVED EARLIER'

driver.execute_script("window.open('');")
driver.switch_to.window(driver.window_handles[1])
driver.get('chrome-extension://{}/popup.html'.format(EXTENSION_ID))
Enter fullscreen mode Exit fullscreen mode

With this Metamask will be opened in this new tab, ready to be interacted with.


That's it, now you're ready to start automating interactions with Dapps.
Follow me for more articles like this and if you have any questions feel free to leave them in the comments below.

Oldest comments (24)

Collapse
 
p12 profile image
P12522 Brasil

Awesome!!!

Collapse
 
bacolen profile image
Raphael Bachofen • Edited

So helpful! Finally someone posted this...

Collapse
 
andrew166 profile image
Andrew

what is use of this ?

Collapse
 
fedeblengio profile image
Federico Blengio

Thanks bro, it has helped me a lot, but I have a problem for days I am trying to solve it but I can not.

any ideas?

Collapse
 
fedeblengio profile image
Federico Blengio

here is my problem
prnt.sc/1oab8yn

Collapse
 
baotran810 profile image
baotran810 • Edited

Hi Federico,
I also got the issue "NoSuchElementException" because it couldn't locate the "Get started" button. While debugging, I found that it focused on the first page when I ran the script, so I switched to the first tab to close then it only focused on the MetaMask page then the "Get started" button was able to locate. Here is the script I used, I hope it would help you.

await driver.getAllWindowHandles().then(function (handles) {
driver.switchTo().window(handles[1]);
driver.close()
driver.switchTo().window(handles[0]);
});

Thread Thread
 
fedeblengio profile image
Federico Blengio

Thank you

Thread Thread
 
fedeblengio profile image
Federico Blengio

I have a question to ask about handles, you can send a dm?

Thread Thread
 
escaraptor09 profile image
Marc Ferri

How did you finally solved it?????

Collapse
 
escaraptor09 profile image
Marc Ferri

I got the same problem

Collapse
 
ltmenezes profile image
Leonardo Teixeira Menezes

Hey Federico, good to know that this article is helping you, thanks for the feedback!
This kinds of issues might happen because the behaviour of the extension might differ depending on the metamask extension version.
In these cases the best approach is to use some python debugger (such as IPDB) to stop the code for you to find exactly what is different in your version, most times you only need to do some minor changes.
Just followed you, feel free to drop a DM if you still have the issue!

Collapse
 
agustinustheo profile image
Agustinus Theodorus

I needed this. Thank you very much!

Collapse
 
techiesworld01 profile image
TechiesWorld01

Hi Leonardo,
I am facing issue while installation of selenium through chrome driver.

ImportError: cannot import name 'webdriver' from partially initialized module 'selenium' (most likely due to a circular import) (C:\Users\Admin\Desktop\selenium.py)

Collapse
 
yournicespice profile image
YourNiceSpice • Edited

Hey Leonardo,I had a problem when I went to the dapp he offered a connection with metamask,
i called
driver.get ('chrome-extension: //nkbihfbeogaeaoehlefnkodbefgpgknn/notification.html#connect/')
i saw connection window
But after that chrome webdriver breaks

Collapse
 
andrew166 profile image
Andrew

Hi,
can you tell me what is profit of this ? anyone explain plz?

Collapse
 
hehe2046 profile image
yoyo

What are your versions of Python and Selenium in the cain?

Collapse
 
f041 profile image
Gabriele Giordano

I get, when trying to input a password in metamask password box:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"#password"}

even though I tried numerous other paths:

Tried by xpath:

//*[@id="password"]

//*[@id="app-content"]/div/div[3]/div/div/form/div/div

('//input')

Tried by ID:

Tried by selector

#password

Collapse
 
cryptotingle profile image
CryptoTingle - 💢 #RouteArmy

Hey guys! If you have problems with the above try this. Just remember to enter your own extension path for metamask (C://...whatever) and your own 12-word seed phrase. Thanks OP! Enjoy!

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

import time

EXTENSION_PATH = 'C:\......INSERT YOUR PATH HERE'

opt = Options()
opt.add_extension(EXTENSION_PATH)

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=opt)

driver.switch_to.window(driver.window_handles[0])
original_window = driver.current_window_handle

Get Started

WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='app-content']/div/div[2]/div/div/div/button"))).click()

Import Wallet

WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='app-content']/div/div[2]/div/div/div[2]/div/div[2]/div[1]/button"))).click()

Donate Data. No thanks!

WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='app-content']/div/div[2]/div/div/div/div[5]/div[1]/footer/button[1]"))).click()

time.sleep(1)

Seed phrase

driver.find_element(By.ID, "import-srp_srp-word-0").send_keys("FIRST WORD")
driver.find_element(By.ID, "import-srp
srp-word-1").send_keys("SECOND WORD")
driver.find_element(By.ID, "import-srp
srp-word-2").send_keys("THIRD WORD")
driver.find_element(By.ID, "import-srp
srp-word-3").send_keys("FOURTH WORD")
driver.find_element(By.ID, "import-srp
srp-word-4").send_keys("FIFTH WORD")
driver.find_element(By.ID, "import-srp
srp-word-5").send_keys("SIXTH WORD")
driver.find_element(By.ID, "import-srp
srp-word-6").send_keys("SEVENTH WORD")
driver.find_element(By.ID, "import-srp
srp-word-7").send_keys("EIGHTH WORD")
driver.find_element(By.ID, "import-srp
srp-word-8").send_keys("NINTH WORD")
driver.find_element(By.ID, "import-srp
srp-word-9").send_keys("TENTH WORD")
driver.find_element(By.ID, "import-srp
srp-word-10").send_keys("ELEVENTH WORD")
driver.find_element(By.ID, "import-srp
_srp-word-11").send_keys("TWELFTH WORD")

Passwords

driver.find_element(By.ID, "password").send_keys("password")
driver.find_element(By.ID, "confirm-password").send_keys("password")

Agree to T&Cs & Submit

WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='create-new-vault__terms-checkbox']"))).click()

Import Button

WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='app-content']/div/div[2]/div/div/div[2]/form/button"))).click()

Congratulations page

WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='app-content']/div/div[2]/div/div/button"))).click()

Intro pop-up box

WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "//*[@id='popover-content']/div/div/section/div[2]"))).click()

Collapse
 
alirio_miguel profile image
Alirio Aranguren

I'm having an issue using Slenium and Java... Whenever I click on the staking button it says that the contract needs to be approved, although it alredy has... I think that it has something to do with the extension but I'm not 100% sure
Image description

Collapse
 
alirio_miguel profile image
Alirio Aranguren

this is what should be displaying
Image description

Collapse
 
aerryasmani profile image
Aerry

Thank you for the articles. Is there anyway that this method can be applied in Katalon Studio as well? Cause I'm new in testing and we're using katalon as a testing tools.

Collapse
 
nmolanog profile image
Nicolas Molano Gonzalez

I am lost with this:
"Click 'Pack extension' and enter the local path to the Metamask extension This will generate a .crx "

When I click Pack extension there is a window which ask for Extension root directory and private key (optional)

I am confused on how this should be done. I need to know where is the local path in my pc where metamask is? if this is the case how can I find this location?

Collapse
 
nmolanog profile image
Nicolas Molano Gonzalez

I was able to find it but I am getting a .pem file instead of an .crx

Collapse
 
berengueradrian profile image
berengueradrian

Thanks a lot for the code. I am having some issues with this approach, I write this comment to see if someone has had a similar issue or can help me with mine.
See this: stackoverflow.com/questions/762522...
And this: dev.to/berengueradrian/runtimecall...

I need to solve this for as much on thursday, it would be really helpfull if someone has the solution, thanks a lot in advance!