Currently working(more like practicing) on data crawling using Python + BeautifulSoup4 to imitate web pages.
It was all fun and games until I figured it is quite challenging to scrape off the web page that uses JavaScript to manipulate the DOM.
Obviously my first approach was to go to google and ask for help and came across this awesome (but with so much confusion) library called Selenium.
I liked the idea of how you can actually ask the code to wait for the browser to load first, and then look for the html elements.
I was able to get some data back but not all the time. Probably needs more studying and playing time with it.
But I feel like this library can be really helpful in the future, mixing it with BS4 could really save my life :)
TL;DR :
- study Selenium
- mix with BS4
- ignore all the copyrights :D
fun fact: I actually researched about Nanoparticle using Selenium lmao
Top comments (0)