DEV Community

HasData profile image Valentina Skakun
Valentina Skakun for HasData

Posted on

Scraping TikTok Profiles and Videos with Python (Part 1)

#tiktok #scraping #python #hasdata

TikTok is a JavaScript-heavy app, and most of the profile data never appears in the static HTML. Even worse, TikTok limits you to about 100 requests per hour per IP, which makes any large-scale scraping almost impossible.

That’s why it’s better to use a web scraping API like HasData.

It handles proxy rotation, bypasses rate limits, and renders JavaScript automatically, so you get the entire page content every time.

Table of Contents

Requirements

You’ll need:

  • Python 3.8+
  • requests
  • beautifulsoup4

A free or paid HasData account to get your API key

Install the dependencies:

pip install requests beautifulsoup4
Enter fullscreen mode Exit fullscreen mode

Get the Page Using HasData API

Instead of making direct requests to TikTok (which trigger rate limits), we’ll send them through HasData.

Each request is routed through a pool of residential proxies, so TikTok sees them as real users from different devices.

import requests
import json
from bs4 import BeautifulSoup

url = "https://www.tiktok.com/@funny_funny66066"

# Get your API key at https://hasdata.com/sign-up
api_key = "YOUR-API-KEY"
api_url = "https://api.hasdata.com/scrape/web"

payload = json.dumps({
    "url": url,
    "proxyType": "residential",
    "proxyCountry": "DE",
    "blockResources": False,
    "wait": 1000,
    "jsRendering": True
})

headers = {
    "Content-Type": "application/json",
    "x-api-key": api_key
}

# Send the request to HasData API
response = requests.post(api_url, headers=headers, data=payload)

# Parse the response as JSON
data = response.json()

# Extract content from the response
content_html = data.get("content")

# Parse HTML
soup = BeautifulSoup(content_html, "html.parser")
Enter fullscreen mode Exit fullscreen mode

At this point, you have a fully rendered TikTok profile page, fetched through a proxy.
No 403 errors, no empty HTML, and no 100-requests-per-hour limits.

Extract Profile Data

TikTok hides most user data inside a <script> block called
__UNIVERSAL_DATA_FOR_REHYDRATION__.
It contains JSON with the entire page content: username, followers, bio, and more.

script = soup.find("script", id="__UNIVERSAL_DATA_FOR_REHYDRATION__")
if not script:
    raise SystemExit("Data script not found. TikTok structure may have changed.")

rehydration_data = json.loads(script.string)

# Extract user details
user_detail = rehydration_data.get("__DEFAULT_SCOPE__", {}).get("webapp.user-detail")

with open("tiktok_user_detail.json", "w", encoding="utf-8") as f:
    json.dump(user_detail, f, ensure_ascii=False, indent=4)

print("Saved user detail data to tiktok_user_detail.json")
Enter fullscreen mode Exit fullscreen mode

This gives you structured JSON data for the user profile, ready to analyze or reuse.

Get Videos from the Profile

Once you have the secUid from the JSON above, you can query TikTok’s internal endpoint /api/post/item_list/ to get the video list.

api_base = "https://www.tiktok.com/api/post/item_list/"
secuid = user_detail["userInfo"]["user"]["secUid"]

params = {
    "aid": "1988",
    "count": "35",
    "cursor": 0,
    "device_platform": "web_pc",
    "secUid": secuid
}

videos_response = requests.get(api_base, params=params)
videos_json = videos_response.json()

with open("tiktok_videos.json", "w", encoding="utf-8") as f:
    json.dump(videos_json, f, ensure_ascii=False, indent=4)
Enter fullscreen mode Exit fullscreen mode

This request will return structured data for each video: description, stats, and direct media URLs.

Bypass Rate Limits and IP Bans

Without proxies, TikTok blocks requests at about 100 per hour. HasData solves this by:

  • Rotating residential IPs automatically
  • Emulating real browser behavior
  • Handling headers and cookies for you

What’s Next

In Part 2, we’ll download the videos you’ve extracted and handle common errors like 403 Access Denied when a download link expires.

Top comments (0)