Get Data from the Web with Web Scraper and Flutter

Intro

There is a huge amount of data online that is accessible with just a few clicks. Web scraping is the process of automating the manual search part with the browser and instead just scraping the info you want from each website. In the tutorial, I will be demonstrating how we can achieve this in Flutter to in the end create an app that displays the world's current population.

Tutorial Portion

The first thing is first, huge shoutout to https://github.com/tusharojha for creating the package https://pub.dev/packages/web_scraper that we will be using for the tutorial

setup

The first thing we have to do is add the package to our flutter project in the pubspec.yaml, The link for the dependency can be found here https://pub.dev/packages/web_scraper#-installing-tab-. Our pubscpec.yaml should look like this:

In main.dart we can delete the MyApp class and we will recreate it as a statless widget that returns a MaterialApp, (optionally already set 'debugShowCheckedModeBanner: false' to not the debug banner):

We will then create our Homescreen which will be a stateful widget, that has a scaffold with a center widget, for now, the widget will just be a circular loading indicator:

Dont forget to set the home as HomeScreen() in main.dart

If we run this there is nothing special just the progress indicator:

Lets now add some variables to keep track of the state of the app; inside _HomeScreenState. We will also create and import our WebScraper now

Then for instantiating all the variables we will first override setState and call an async method that we created called _getData(). We will handle all the asynchronous calls in _getData()

We will now build out the _getData() method: we need to create a webscraper with our desired wesite url (in our case 'https://worldpopulationreview.com'). Then we want to check if it can load our desired page (since our website has the data in the main page this will just be '/').

HTML Time !!

Now we can get what we want from the webpage. We need to first open the page in a browser and inpect the elements (in chrome cntl+alt+J)

We can get the element's name from the first options in properties. For the one which we want it is a div of the class center so our name will be div.center. We create a list of maps to get all the elements that match this name:

Once we have that we can call setState and set our popNum that we defined earlier to the first element's title that we get and set the bool loaded to true so the whole method should look now like:

Bringing it all Together

The last thing we need to do now is stop the progress indicator when we get the result and display the popNum in a Text widget instead. We can do this by using the dart ternary operation to check if the loaded bool is true, and show the Text if it is true.

So now our app should open with the loading indicator and then show the text we scraped once it is ready

Conclusion

Now we can easily scrape any static elements from websites with HTML. This is particularly useful if we want to, for example, compare the first price result of each seller's website and show them all, we would have to create a getData method for each site we are scraping then return it in a list.

For the souce code: souce