Discussion on: My Top 5 Awesome Python Projects

[Juan Carlos]

Web scraper alternative:

juancarlospaco / faster-than-requests

Faster requests on Python 3

Faster-than-Requests

Library	Speed	Files	LOC	Dependencies	Developers	Multi-Threaded Web Scraper Built-in
PyWGET	152.39	1	338	Wget	>17	❎
Requests	15.58	>20	2558	>=7	>527	❎
Requests (cached object)	5.50	>20	2558	>=7	>527	❎
Urllib	4.00	???	1200	0 (std lib)	???	❎
Urllib3	3.55	>40	5242	0 (No SSL), >=5 (SSL)	>188	❎
PyCurl	0.75	>15	5932	Curl, LibCurl	>50	❎
PyCurl (no SSL)	0.68	>15	5932	Curl, LibCurl	>50	❎
Faster_than_requests	0.45	1	99	0	1	✔️ 6, One-Liner

• Lines Of Code counted using CLOC.
• Direct dependencies of the package when ready to run.
• Benchmarks run on Docker from Dockerfile on this repo.
• Developers counted from the Contributors list of Git.
• Speed is IRL time to complete 10000 HTTP local requests.
• Stats as of year 2019.
• x86_64 64Bit AMD, SSD, Arch Linux.

Use

import faster_than_requests as requests
requests.get("http://httpbin.org/get")                             # GET
requests.post("http://httpbin.org/post

…

View on GitHub

Faster Than Requests with Multi-Thread URL-to-SQLite Web Scraper

Juan Carlos ・ Dec 24 '19 ・ 1 min read

#python