Flutter Web Scraper Level 2: Etsy Product Finder Part 2

Intro

I am showing how to build a etsty product searcher utilizing webscraping in flutter. In the previous part (link here) I showed how to set the ui, and we ended with this

Our goal for this part is to add the url build to make an etsy search and the webscraping of the search results. In the end we will have an app that looks like this:

Tutorial Portion

'Please enter search' text

The first thing we will add is a loading indicator for when we are not searching. For this

in home_screen.dart we add:

	@override
	Widget build(BuildContext context) {
	return Scaffold(
	resizeToAvoidBottomInset: false,
	backgroundColor: Colors.white,
	body: Container(
	width: MediaQuery.of(context).size.width,
	height: MediaQuery.of(context).size.height,
	child: SafeArea(
	child: Column(
	children: [
	SearchBar(homeScreen: widget,),
	(searched)
	? Expanded(
	child: ResultList(
	search: search,
	),
	)
	: Padding(
	padding: EdgeInsets.symmetric(vertical: 100),
	child: Text('Please enter a search',),
	),
	],
	),
	),
	),
	);
	}

view raw home_screen.dart hosted with ❤ by GitHub

Webscraping Time

In results_list.dart we are going to add the changes to scrape the products from Esty's page. I will just add the code for the new result_list.dart file and explain some of the most important lines bellow:

	class ResultList extends StatefulWidget {
	final String search;
	const ResultList({Key key, this.search}) : super(key: key);
	@override
	_ResultListState createState() => _ResultListState();
	}
	class _ResultListState extends State<ResultList> {
	bool loaded = false;
	List<ResultItem> resultList = List();
	String etsyUrl = 'https://www.etsy.com';
	String pageEtension = '/market/';
	@override
	void initState() {
	super.initState();
	getData();
	}
	void getData() async {
	final webScraper = WebScraper(etsyUrl);
	print(webScraper.getPageContent());
	if (await webScraper.loadWebPage(
	'$pageEtension${widget.search.trim().replaceAll(' ', '%20')}')) {
	List<Map<String, dynamic>> images = webScraper.getElement(
	'img.width-full.wt-height-full.display-block.position-absolute',
	['src']);
	List<Map<String, dynamic>> descriptions = webScraper.getElement(
	'h3.text-gray.text-truncate.mb-xs-0.text-body', ['title']);
	List<Map<String, dynamic>> etsyPrices =
	webScraper.getElement('span.currency-value', ['innerHtml']);
	List<Map<String, dynamic>> urls = webScraper.getElement(
	'div > ul > li > div > a',
	['href', 'title']);
	images.forEach((image) {
	int i = images.indexOf(image);
	resultList.add(
	ResultItem(
	image: images[i]['attributes']['src'],
	description: descriptions[i]['title'].toString().trim(),
	url: urls[i]['attributes']['href'],
	price: etsyPrices[i]['title'],
	),
	);
	});
	setState(() {
	loaded = true;
	});
	}
	}
	@override
	Widget build(BuildContext context) {
	return Scaffold(
	body: Center(
	child: (loaded)
	? ListView(
	physics: BouncingScrollPhysics(),
	children: resultList.getRange(0, 5).toList(),
	)
	: CircularProgressIndicator(),
	),
	);
	}
	}

view raw result_list.dart hosted with ❤ by GitHub

First, in lines 11-12, we have a boolean to check if the webscraping has loaded, and a list of result items that we will be adding to the list.

Lines 14 is the the main Estys url and line 15 is the particular page we will be searching.

We have an initState function that calls the async getData() function.

getData() is the function where we will be scraping the page. First we initialize a new WebScraper variable that has the estyUrl. Then in an if statement we check if the webpage we chose gets loaded. For the webpage in line 27, you can see that we have to construct the URL, by adding the search onto our pageExtenstion. The '.trim().replaceAll(' ', '%20')' portion is to make sure that if we have spaces in our search we are trimming them and changing any space between words to '%20' which is the standard that Etsy uses.

Then we create three List> to get the images, descriptions, prices, and URLs. The tags used in the getElement() call were found using the web inspector in chrome on the Etsy page

We the iterate through one of the lists (in this case I chose images) and custruct our ResultItem then add it to the list.

Once all this is done we call setState() and set loaded to true so that the results list will be show.

And that's it!

Now if you run it you should get the end product

Conclusion

I hope this projust tought you how to search and scrape html pages to display them in a list to users. This is a great idea if you want to create a comparison searcher, you could then scrape two websites and display all the result.

In a future post I will also be showing how to scrape Javascript website, something that is a bit more tricky and requires more setting up, so stay tuned to for that one if you are interested.