Flutter Web Scraper Level 2: Etsy Product Finder Part 1

Intro

In my previous post (post) I showed how we can use web scraper to get the current world population number and display, however, this is a pretty simple and useless idea. So no I want to show how to use web scraper to make an app that is more like what you would be used to seeing. The application is called Etsy Product Finder and it will allow users to search for a product and view the top 5 Etsy results with their prices and if the user wants to view the product on the Etsy website, a clickthrough will be implemented. Since this will be a long walk through I will be posting 2 parts, the first one will cover installing all the packages and setting up the ui. The second will focus on how to use the data by web scraping the etsy website.

Tutorial Portion

Installing Packages

The first thing we need to do is add the packages that we need to our pubspec yaml. In this case it will be web_scraper and url_launcher, these packages allow us to do the necessary scraping data and also then allowing the user to clickthrough to open the item's page in the browser

	dependencies:
	flutter:
	sdk: flutter
	web_scraper: any
	url_launcher: ^5.4.11

view raw pubspec.yaml hosted with ❤ by GitHub

Then in main.dart you do the usual and clear everything out leaving only the main function and a stateless widget that has it's home being you homescreen:

	void main() {
	runApp(MyApp());
	}
	class MyApp extends StatelessWidget {
	const MyApp({Key key}) : super(key: key);
	@override
	Widget build(BuildContext context) {
	return MaterialApp(
	debugShowCheckedModeBanner: false,
	home: HomeScreen(),
	);
	}
	}

view raw main.dart hosted with ❤ by GitHub

Now I will first breakdown what we are going to build in the first part:

• Main Screen with a search bar at the top
• List to contain search result
• List items for the result list

In this part we will be going through only the UI elements of the app

Creating Homescreen

We need to create a homescreen with a search bar at the top and a list underneath it.

So we are first going to have

	class HomeScreen extends StatefulWidget {
	_HomeScreenState state = _HomeScreenState();
	@override
	_HomeScreenState createState() => state;
	}
	class _HomeScreenState extends State<HomeScreen> {
	bool searched = false;
	String search = '';
	@override
	void initState() {
	super.initState();
	}
	@override
	Widget build(BuildContext context) {
	return Scaffold(
	resizeToAvoidBottomInset: false,
	backgroundColor: Colors.white,
	body: Container(
	width: MediaQuery.of(context).size.width,
	height: MediaQuery.of(context).size.height,
	child: SafeArea(
	child: Column(
	children: [
	SearchBar(homeScreen: widget,),
	],
	),
	),
	),
	);
	}
	}

view raw home_screen.dart hosted with ❤ by GitHub

This create a homescreen with a scaffold within a safe area (widget that avoid notches and notification bar), and it has a column, whose only child is the appbar currently.

_HomeScreen state contains the variables for if the search was has been made and the string for the search

As you will notice in the Homescreen class we create _HomeScreenState as a variable (state) then pass it on to createState(), this is because we are going to want to be able to change the state from the search bar class. which is what we will be doing next.

Creating the Seachbar

For the search bar, we will use a row that contains a texform field and a material button that will trigger the search.

	class SearchBar extends StatelessWidget {
	TextEditingController controller = TextEditingController();
	final HomeScreen homeScreen;
	SearchBar({ this.homeScreen});
	getSearch(){
	return controller.text;
	}
	@override
	Widget build(BuildContext context) {
	return Container(
	color: Colors.orange.shade300,
	child: Row(
	mainAxisAlignment: MainAxisAlignment.center,
	children: [
	Padding(
	padding: const EdgeInsets.symmetric(horizontal: 20.0, vertical: 40.0),
	child: Container(
	width: 200.0,
	child: TextFormField(
	controller: controller,
	decoration: InputDecoration(
	filled: true,
	fillColor: Colors.black12,
	hintText: 'enter search',
	hintStyle: TextStyle(
	color: Colors.white38
	)
	),
	style: TextStyle(
	color: Colors.white70
	),
	),
	),
	),
	MaterialButton(
	child: Text('Search', style: TextStyle(color: Colors.white70),),
	onPressed: (){
	homeScreen.state.setState(() {
	homeScreen.state.searched = true;
	homeScreen.state.search = controller.text;
	});
	},
	)
	],
	),
	);
	}
	}

view raw search_bar.dart hosted with ❤ by GitHub

In the code above you can see that we pass an instance of homescreen to the search bar, in turn when the material button is pressed, we change the homescreen's state so that searched is true and, the search text is the text from the textformfeild.

Next we will design the result item and the result list.

Result Item

The result item has to include variables for:

• String for item's image url
• String for item's description
• String for item's clickthrough url
• String for item's prices

	class ResultItem extends StatelessWidget {
	final String image;
	final String url;
	final String description;
	final String price;
	const ResultItem(
	{Key key, this.image, this.url, this.description, this.price})
	: super(key: key);
	@override
	Widget build(BuildContext context) {
	return Padding(
	padding: const EdgeInsets.symmetric(vertical: 10.0),
	child: GestureDetector(
	onTap: ()=> _launchUrl(),
	child: Container(
	height: 400,
	width: 300,
	child: Column(
	mainAxisSize: MainAxisSize.min,
	mainAxisAlignment: MainAxisAlignment.spaceBetween,
	crossAxisAlignment: CrossAxisAlignment.center,
	children: [
	ClipRRect(
	borderRadius: BorderRadius.circular(15.0),
	child: Image.network(
	image,
	height: 300,
	width: 300,
	fit: BoxFit.fitHeight,
	),
	),
	Text(
	description ?? 'failed to load',
	style: TextStyle(
	color: Colors.black87,
	fontSize: 18.0,
	fontWeight: FontWeight.w300,
	),
	overflow: TextOverflow.ellipsis,
	textAlign: TextAlign.center,
	),
	Text(
	price ?? 'failed to load',
	style: TextStyle(
	color: Colors.black87,
	fontSize: 24.0,
	fontWeight: FontWeight.w500,
	letterSpacing: 1.2,
	),
	),
	],
	),
	),
	),
	);
	}
	_launchUrl() async {
	await launch(url);
	}
	}

view raw result_item.dart hosted with ❤ by GitHub

For the text we use a null checker to make sure we are not passing null results. The whole thing is wraped in an GestureDetector that calls _lauchUrl(). This in turn ia an async method that calls launch(url) to open the clickthough link.

Now to create a list of these

Result List

For testing puproses we won't webscrape yet, instead for this part we generate 5 test list items just to see what it looks like.

	class ResultList extends StatefulWidget {
	final String search;
	const ResultList({Key key, this.search}) : super(key: key);
	@override
	_ResultListState createState() => _ResultListState();
	}
	class _ResultListState extends State<ResultList> {
	@override
	Widget build(BuildContext context) {
	return ListView(
	children: List.generate(
	5,
	(index) => ResultItem(
	description: 'test Image',
	image: 'https://via.placeholder.com/150',
	url: 'https://via.placeholder.com/150',
	price: '\$500',
	),
	).toList(),
	);
	}
	}

view raw result_list.dart hosted with ❤ by GitHub

Then in test_screen.dart we can add to the column:

(searched)?Expanded(child: ResultList(search: 'test',)):Placeholder(color: Colors.transparent,),

view raw home_screen.dart hosted with ❤ by GitHub

if we run this and click search we will see:

Conclusion

In the next part I will show how we can add the webscraping to scrape search results from etsy

source code