Dev.to Scraper made of Telegram 🌚

Illia Pyshniak — Fri, 23 Aug 2019 18:47:44 +0000

I love when there is no need to wait browser/app open to read new aricle. And I love automation.

So I created DEV Telegram Channel, where you can read dev.to articles. And you can do it INSTANTLY

Comparison: Browser | App | Instant View in Telegram

What is Instant View?

Instant View is a built-in Telegram tool which allow to read articles without huge loadtime.

Main concept of IV is a Template. It tells Telegram severs what data to cache and what to remove.

So all process is simple:

Article send via Telegram
Telegram check for Template
Telegram scraps and caches article
Pure-HTML article provided to user

What is Template?

Template is a code, which tell Telegram scraper what to do with page.

IV Template Language based on:

YAML syntax
XPath to select nodes
RegEx to... RegExing 🌚

This code is enough to scrap 50% articles on dev.to:

~version: "2.1"
body:     //div[@id="article-body"] # Use <div id="article-body"> as main article
title:    //h1[0] # First h1 header on page used as title

Yes, 3 lines of code.

How to use your Template?

Once you have finished your code on My Templates page, press "View in Telegram" button. You would get something like that:

Now you can create IV for any article using this template: t.me/iv?url=LINK&rhash=XXX

So now we have 2 goals:

Automate posting
Make a pretty post ~~and get rid of this ugly long link~~

What is IFTTT?

IFTTT ("if this, then that") - service, which allows you to connect different services via Applets. Applet react on Trigger and respond with Action (e.g. post in Telegram with new RSS article)

All you need is create applet on IFTTT Platform.

Hint how to make all look pretty

You can hide all ugly links in '⁠' - narrow non-breaking space. Yes, there is a character between the quotes.

And as a result - a piece of perfectness:

Epilogue

Article ends, so if you don't know what to do next, you can consider:

Subscribe to DEV.TO Telegram channel that I've made or
Use IFTTT Applet to send brand new dev.to articles to your channel/group
Check gist with overdocumented IV Template (66 LOC)
Send issues if there is any problems in generated articles (missing images, text etc.)
Become a Patron, so I can buy a pizza (I love pizza! 🍕👋🏻(👀)👍🏻)

DEV Community: Illia Pyshniak