Web Scraping Tutorial - Elixir API Server with a Chrome Extension

Eduardo Alencar — Fri, 17 Jun 2022 23:40:06 +0000

Imagine a site cars classification and You should follow a change the prices the determinate Vehicle Models. I called the fake site ‘FakeCars’, the site with a lot system anti-bots systems. Difficult to use Web Scraping default, but it doesn't impossible!

We choose models for price tracking with the idea of creating a Chrome Extension to reading the total number of ads for the page and sending for our API Server. We analyze their prices week-by-week.

Using stack for Price Tracking:

API server in Elixir with Phoenix framework — Coding in next post, in then moment our using the URL for API server: "http://localhost:4000/api/v1/chrome_ads/"
Chrome Extension in JavaScript

Coding Chrome Extension

Structure for a chrome extension

Our code
manifest.json v.3

js/background.js

js/popup.js

js/fakecars_contentscript.js

The JSON structure sent to API Server a for each page read.

Coding API server

1 - Creating the Phoenix app

2 - Configure in config/dev.exs your PostgreSQL login.

Creating ad model

$ mix phx.gen.schema Vehicle.Ad vehicle_ads ad_date:utc_datetime ad_id:string ad_source:string city:string km:integer price:integer state:string vehicle_brand:string vehicle_model:string vehicle_version:string year_manufacture:integer year_model:integer active:boolean

Create the database and run migrate

$ mix ecto.create
$ mix ecto.migrate

Configure in 'lib/ads_api_web/route.ex' :
Finally, we create api_controller.ex in 'lib/ads_api_web/controllers/' for consuming the JSON received and saving it in the database.

DEV Community: Eduardo Alencar

Web Scraping Tutorial - Elixir API Server with a Chrome Extension

Coding Chrome Extension

Coding API server