DEV Community

Cover image for Get Webpage with python requests
Waylon Walker
Waylon Walker

Posted on • Originally published at waylonwalker.com

6 3

Get Webpage with python requests

Python's requests library is one of the gold standard apis, designed by Kenneth Reitz. It was designed with the user perspective in mind first and implementation second. I have heard this called readme driven development, where the interface the user will use is laid out first, then implemented. This makes the library much mor intuitive than if it were designed around how it was easiest to implement.

Install Requests

Requests is on pypi and can be installed into your virtual environtment with pip.

python -m pip install requests
Enter fullscreen mode Exit fullscreen mode

Getting the content of a request

Requests makes getting content from a web url as easy as possible.

import requests

r = requests.get('https://waylonwalker.com/til/htmx-get/') 
article = r.content
Enter fullscreen mode Exit fullscreen mode

requests is not limited to html

Requests can handle any web request and is not limited to only html. Here are some examples to get a markdown file, a csv, and a png image.

htmx_get_md = requests.get('https://waylonwalker.com/til/htmx-get.md').content
cars = requests.get('https://waylonwalker.com/cars.csv').content 
profile = requests.get('https://images.waylonwalker.com/8bitc.png').content
Enter fullscreen mode Exit fullscreen mode

RTFM

There is way more to requests, this just scratches the surface while covering what you are going to need to get going. The requests docs have way more details.

Postmark Image

Speedy emails, satisfied customers

Are delayed transactional emails costing you user satisfaction? Postmark delivers your emails almost instantly, keeping your customers happy and connected.

Sign up

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

👋 Kindness is contagious

Immerse yourself in a wealth of knowledge with this piece, supported by the inclusive DEV Community—every developer, no matter where they are in their journey, is invited to contribute to our collective wisdom.

A simple “thank you” goes a long way—express your gratitude below in the comments!

Gathering insights enriches our journey on DEV and fortifies our community ties. Did you find this article valuable? Taking a moment to thank the author can have a significant impact.

Okay