Web Scraping Tutorial - Elixir API Server with a Chrome Extension

Imagine a site cars classification and You should follow a change the prices the determinate Vehicle Models. I called the fake site ‘FakeCars’, the site with a lot system anti-bots systems. Difficult to use Web Scraping default, but it doesn't impossible!

We choose models for price tracking with the idea of creating a Chrome Extension to reading the total number of ads for the page and sending for our API Server. We analyze their prices week-by-week.

Using stack for Price Tracking:

• API server in Elixir with Phoenix framework — Coding in next post, in then moment our using the URL for API server: "http://localhost:4000/api/v1/chrome_ads/"

• Chrome Extension in JavaScript

Coding Chrome Extension

Structure for a chrome extension

Our code
manifest.json v.3

js/background.js

js/popup.js

js/fakecars_contentscript.js

The JSON structure sent to API Server a for each page read.

Coding API server

1 - Creating the Phoenix app

2 - Configure in config/dev.exs your PostgreSQL login.

• Creating ad model

$ mix phx.gen.schema Vehicle.Ad vehicle_ads ad_date:utc_datetime ad_id:string ad_source:string city:string km:integer price:integer state:string vehicle_brand:string vehicle_model:string vehicle_version:string year_manufacture:integer year_model:integer active:boolean

• Create the database and run migrate

$ mix ecto.create
$ mix ecto.migrate

• Configure in 'lib/ads_api_web/route.ex' :
  

• Finally, we create api_controller.ex in 'lib/ads_api_web/controllers/' for consuming the JSON received and saving it in the database.