DEV Community

DailyFlutter.Monster
DailyFlutter.Monster

Posted on • Originally published at dailyflutter.monster

1 1

Flutter Web Scraper Level 2: Etsy Product Finder Part 2

Intro

I am showing how to build a etsty product searcher utilizing webscraping in flutter. In the previous part (link here) I showed how to set the ui, and we ended with this

Alt Text

Our goal for this part is to add the url build to make an etsy search and the webscraping of the search results. In the end we will have an app that looks like this:

Alt Text

Tutorial Portion

'Please enter search' text

The first thing we will add is a loading indicator for when we are not searching. For this

in home_screen.dart we add:

@override
Widget build(BuildContext context) {
return Scaffold(
resizeToAvoidBottomInset: false,
backgroundColor: Colors.white,
body: Container(
width: MediaQuery.of(context).size.width,
height: MediaQuery.of(context).size.height,
child: SafeArea(
child: Column(
children: [
SearchBar(homeScreen: widget,),
(searched)
? Expanded(
child: ResultList(
search: search,
),
)
: Padding(
padding: EdgeInsets.symmetric(vertical: 100),
child: Text('Please enter a search',),
),
],
),
),
),
);
}

Webscraping Time

In results_list.dart we are going to add the changes to scrape the products from Esty's page. I will just add the code for the new result_list.dart file and explain some of the most important lines bellow:

class ResultList extends StatefulWidget {
final String search;
const ResultList({Key key, this.search}) : super(key: key);
@override
_ResultListState createState() => _ResultListState();
}
class _ResultListState extends State<ResultList> {
bool loaded = false;
List<ResultItem> resultList = List();
String etsyUrl = 'https://www.etsy.com';
String pageEtension = '/market/';
@override
void initState() {
super.initState();
getData();
}
void getData() async {
final webScraper = WebScraper(etsyUrl);
print(webScraper.getPageContent());
if (await webScraper.loadWebPage(
'$pageEtension${widget.search.trim().replaceAll(' ', '%20')}')) {
List<Map<String, dynamic>> images = webScraper.getElement(
'img.width-full.wt-height-full.display-block.position-absolute',
['src']);
List<Map<String, dynamic>> descriptions = webScraper.getElement(
'h3.text-gray.text-truncate.mb-xs-0.text-body', ['title']);
List<Map<String, dynamic>> etsyPrices =
webScraper.getElement('span.currency-value', ['innerHtml']);
List<Map<String, dynamic>> urls = webScraper.getElement(
'div > ul > li > div > a',
['href', 'title']);
images.forEach((image) {
int i = images.indexOf(image);
resultList.add(
ResultItem(
image: images[i]['attributes']['src'],
description: descriptions[i]['title'].toString().trim(),
url: urls[i]['attributes']['href'],
price: etsyPrices[i]['title'],
),
);
});
setState(() {
loaded = true;
});
}
}
@override
Widget build(BuildContext context) {
return Scaffold(
body: Center(
child: (loaded)
? ListView(
physics: BouncingScrollPhysics(),
children: resultList.getRange(0, 5).toList(),
)
: CircularProgressIndicator(),
),
);
}
}

First, in lines 11-12, we have a boolean to check if the webscraping has loaded, and a list of result items that we will be adding to the list.

Lines 14 is the the main Estys url and line 15 is the particular page we will be searching.

We have an initState function that calls the async getData() function.

getData() is the function where we will be scraping the page. First we initialize a new WebScraper variable that has the estyUrl. Then in an if statement we check if the webpage we chose gets loaded. For the webpage in line 27, you can see that we have to construct the URL, by adding the search onto our pageExtenstion. The '.trim().replaceAll(' ', '%20')' portion is to make sure that if we have spaces in our search we are trimming them and changing any space between words to '%20' which is the standard that Etsy uses.

Then we create three List> to get the images, descriptions, prices, and URLs. The tags used in the getElement() call were found using the web inspector in chrome on the Etsy page

Alt Text

We the iterate through one of the lists (in this case I chose images) and custruct our ResultItem then add it to the list.

Once all this is done we call setState() and set loaded to true so that the results list will be show.

And that's it!

Now if you run it you should get the end product

Alt Text

Conclusion

I hope this projust tought you how to search and scrape html pages to display them in a list to users. This is a great idea if you want to create a comparison searcher, you could then scrape two websites and display all the result.

In a future post I will also be showing how to scrape Javascript website, something that is a bit more tricky and requires more setting up, so stay tuned to for that one if you are interested.

Sentry mobile image

Mobile Vitals: A first step to Faster Apps

Slow startup times, UI hangs, and frozen frames frustrate users—but they’re also fixable. Mobile Vitals help you measure and understand these performance issues so you can optimize your app’s speed and responsiveness. Learn how to use them to reduce friction and improve user experience.

Read the guide →

Top comments (0)

Sentry mobile image

Improving mobile performance, from slow screens to app start time

Based on our experience working with thousands of mobile developer teams, we developed a mobile monitoring maturity curve.

Read more

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay