DEV Community

Kacper Włodarczyk
Kacper Włodarczyk

Posted on • Edited on

2 1 1 1

[Python] How to get full url from shortened url

Sometimes when you scrape a website, you may have encountered the fact that the website returns shortened URLs to sources from other websites.

As in this case, for example, https://upflix.pl/r/Qb64Ar this link consists of a domain and some random characters. The way a shortened link works is that it redirects you to another page. Therefore, the status_code that our query returns is 302

Sometimes it happens that we need a full URL to get that can do this with a few lines of Python code and the requests library.

pip install requests
Enter fullscreen mode Exit fullscreen mode

We will use the head method to perform this function
This method is similar to get with the difference that it does not return any content, only headers.

response = requests.head(short_url)
Enter fullscreen mode Exit fullscreen mode

After executing the query, we can check the headers that were returned.
There is information here such as:

  • date
  • type of website content
  • character encoding
  • FULL LINK and many other information you can see below.
{'Date': 'Thu, 16 Nov 2023 00:43:13 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Connection': 'keep-alive', 'location': 'https://www.imdb.com/title/tt14060708/', 'vary': 'Origin', 'x-powered-by': 'PHP/7.3.33', 'x-frame-options': 'SAMEORIGIN', 'CF-Cache-Status': 'DYNAMIC', 'Report-To': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=bgvCMcMQg1ZkjanlgqzemKUHHthalhb%2FAT72Q58O8a22eFmkeb%2FyeeIMfKkGFwt8WmkMB6dv28F1G2CdH134Kilk%2BcdQNweIZ3O%2FN9KlQf1A2VF%2Bm3yYT89rvjU%3D"}],"group":"cf-nel","max_age":604800}', 'NEL': '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}', 'Strict-Transport-Security': 'max-age=15552000; includeSubDomains; preload', 'X-Content-Type-Options': 'nosniff', 'Server': 'cloudflare', 'CF-RAY': '826bb2ce597ebfda-WAW', 'alt-svc': 'h3=":443"; ma=86400'}
Enter fullscreen mode Exit fullscreen mode

Full code

import requests
from typing import Optional 

def get_full_url(short_url: str) -> Optional[str]
    response = requests.head(short_url)
    if response.status_code == 302:
        headers = response.headers
        return headers["location"]

    return None 

Enter fullscreen mode Exit fullscreen mode

AWS Security LIVE!

Join us for AWS Security LIVE!

Discover the future of cloud security. Tune in live for trends, tips, and solutions from AWS and AWS Partners.

Learn More

Top comments (2)

Collapse
 
m0n0x41d profile image
Ivan Zakutnii
import requests

full_url = requests.get("https://shorturl.at/cikuPxx").url
Enter fullscreen mode Exit fullscreen mode

It will return either the full URL or an alternative string, depending on the shortener provider. Typically, this alternative string is the shortener service URL if the link is broken.

Collapse
 
axsddlr profile image
Recluse • Edited

missing optional import from typing and colon after optional

from typing import Optional
import requests
def get_full_url(short_url: str) -> Optional[str]:

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs

👋 Kindness is contagious

Dive into an ocean of knowledge with this thought-provoking post, revered deeply within the supportive DEV Community. Developers of all levels are welcome to join and enhance our collective intelligence.

Saying a simple "thank you" can brighten someone's day. Share your gratitude in the comments below!

On DEV, sharing ideas eases our path and fortifies our community connections. Found this helpful? Sending a quick thanks to the author can be profoundly valued.

Okay