DEV Community: Nicholas Bennett

Promptagogy: Beyond Prompt Engineering

Nicholas Bennett — Thu, 01 May 2025 18:35:17 +0000

Written by Nicholas Bennett using AI tools including Claude.ai and ChatGPT for editorial refinement

TL;DR: Prompt engineering focuses on crafting perfect prompts, but there's equal value in learning from how others converse with AI. I'm introducing "promptagogy": the practice of metacognitive learning through AI interaction that prioritizes the observation and documentation of diverse questioning pathways, treating prompts as windows into human thought processes rather than merely as tools for generating specific outputs. Join the waitlist for promptagogy.com to be part of this complementary approach.

A New Approach to AI Literacy

While countless courses teach formulas for crafting perfect prompts, they often overlook an equally powerful approach to AI literacy: learning by watching how others interact with these systems. The prompt engineering approach, valuable as it is, has an important counterpart that I call promptagogy.

Promptagogy is the practice of metacognitive learning through AI interaction that prioritizes the observation and documentation of diverse questioning pathways, treating prompts as windows into human thought processes rather than merely as tools for generating specific outputs.

The term draws inspiration from "pedagogy," derived from the Greek "paidagōgos," literally meaning "to lead the child" where "país" means child and "ágō" means to lead or guide. While traditional pedagogy emerged as the art and science of teaching, promptagogy applies these educational principles to the evolving practice of AI interaction. Just as pedagogy evolved from its ancient Greek origins to a sophisticated discipline of educational theory and practice, promptagogy represents a natural evolution. It shifts from merely crafting effective prompts to understanding the deeper learning processes that occur when humans engage with AI through conversation.

The Complement to Current Approaches

Prompt engineering courses treat AI interaction like a programming language: follow specific syntax rules, use particular keywords, structure your request in the "correct" way.

This approach works. It's valuable. But it's only half the picture. Here's what it often misses:

Diverse thinking styles. Rule-following, structured approaches work for some minds but aren't intuitive for everyone.
The evolution process. Models and techniques change. Understanding the process of refining prompts is as important as knowing the current best practices.
The learning journey. You see the final, optimized prompt but not the path taken to discover it.

What's needed isn't a replacement for prompt engineering, but a complementary approach that embraces the messiness and diversity of how humans naturally learn.

Promptagogy in Action: Real-World Examples

Consider these simple conversation evolutions I've collected. Notice how each second prompt starts with a quick "pause" that acknowledges the AI's response, then narrows the request to something more practical:

First (general) prompt	Second (refined) prompt
"Can you explain what a black hole is?"	"That's too much information; could you give me one simple analogy a teenager would get?"
"What are the basics of healthy eating?"	"I don't understand. Can you boil it down to three easy tips I can start today?"
"What exactly is happening with climate change?"	"That's not what I wanted. Just tell me two everyday actions that genuinely make a difference."
"How do people switch careers into tech?"	"That's too much information. What's the very first step I should take this month?"
"What should I see if I visit Paris?"	"I don't understand. Can you suggest one must-see place and the best time of day to go?"

These examples reveal a pattern: effective prompt refinement often involves acknowledging where the first response fell short, then narrowing scope dramatically toward concrete, actionable information. This evolution from broad to specific, from overwhelming to manageable, represents a microcosm of the learning process itself.

From Technical Complexity to Vivid Understanding

More technical domains benefit even more dramatically from prompt refinement. Notice how these second prompts specifically request analogies, metaphors, and examples to transform abstract concepts into vivid understanding:

Topic	First (introductory) prompt	Second (refined) prompt
Black holes	"What exactly are black holes and how do they form?"	"That's fascinating but abstract. Can you give me a kitchen-based metaphor for how a black hole's gravity works that I could explain to my 12-year-old?"
Large-scale farming	"How does industrial agriculture work in modern times?"	"I'm still confused. Could you use an analogy comparing traditional farming to modern precision agriculture, like how a craftsman compares to something else?"
Job hunting	"What's the best way to get a job in today's market?"	"That feels overwhelming. Can you give me a concrete example of how one specific person successfully navigated a career change into tech with no experience?"
US economy	"How does the Federal Reserve impact the US economy?"	"That's too technical. Could you use a metaphor involving a thermostat in a house to explain how the Fed controls inflation?"
Language learning	"What are the most effective methods for learning a new language?"	"I need something more tangible. Can you give me an analogy comparing language learning to building a physical skill, with specific examples of what the 'practice drills' would look like?"

In technical domains especially, we see how the second prompts don't just narrow scope. They transform abstract information into concrete understanding through deliberate requests for analogies and examples. This pattern of refinement is particularly powerful for topics that are naturally abstract or complex.

What fascinates me is that different people will refine their prompts in different ways, reflecting their unique cognitive approaches. Some people naturally break problems into sequential steps, while others prefer concrete examples first. Some navigate by metaphor, others by clear definitions. There's no single "correct" way to evolve a conversation with AI.

My Journey to Promptagogy

Early Learning Experiences

I've never resonated with spoon-fed approaches to learning. School's rigid structures and predetermined pathways left me feeling constrained, not inspired. I've always learned best through examples and my own explorations. I discover patterns. I test boundaries. I find my own path to understanding.

At 13, I taught myself C and assembly language without textbooks or formal courses. I talked with friends and bounced ideas off each other. We downloaded code from programmers in Europe. We examined it line by line. We experimented with modifications. We learned through this organic process of exploration.

Looking at code, especially seeing how it evolved through iterations, became a window into someone else's crystallized thinking. It offered a way to understand not just the subject of the code but the very practice of transforming thought into prescription.

Professional Evolution

For 16 years, I worked in desktop tech support where I developed a practice that would later influence my thinking about promptagogy. I became a dedicated self-documentarian in my support tickets. I wrote down everything I tried and failed, along with the final solution. This served multiple purposes: it helped me reason out loud, share work with others, learn from my own reasoning journey, and invite colleagues to learn from my process as well. This practice of documenting the full troubleshooting journey helped our entire team become better problem solvers.

I also learned the importance of adaptive communication. When explaining technical issues to users, I discovered that starting with questions rather than answers was far more effective. For instance, if the Chief of Staff asked what I did to make her laptop faster, I wouldn't launch into a monologue about SSD performance and Windows optimization. Instead, I'd ask, "Have you ever used the Task Manager to close a frozen program?" This simple question would help me gauge her technical vocabulary and understanding.

If she wasn't familiar with Task Manager, I'd use an analogy: "I made your laptop faster by returning it to a clean state with your data intact. Computers just need periodic tune-ups like cars." But if she demonstrated more technical knowledge, I could build on that foundation with more detailed explanations. This question-first approach allowed me to adapt my explanation to her existing mental model rather than forcing her to adapt to mine.

Finding Models for Adaptive Learning

When mentoring junior technicians, I approached them like users of a product that I needed to empower. This led me to Kathy Sierra's book "Badass: Making Users Awesome," where she cites studies showing we learn many times faster by watching an expert perform a task than from simply receiving textbook bottom-up instruction. She referenced research by Phillip Kellman from UCLA and Mary Kaiser from NASA presented in 1994 that found: "Within two hours, non-pilots were faster at accurately interpreting the cockpit instruments than seasoned pilots with an average of more than 1,000 hours flight experience." Seeing the process, including the mistakes and corrections, accelerated learning in ways that perfect demonstrations couldn't match.

I've long felt a kinship with young King Arthur learning from Merlin in "The Sword in the Stone." I dreamed of having my own knowledgeable wizard to question.

I also had Penny's Computer Book on my wishlist from Inspector Gadget; a responsive, intelligent guide that provided just the information needed at just the right time.

These weren't just fantasy escapism. They represented an ideal learning relationship: responsive, adaptive, patient, and tailored to the individual learner. Now, by engaging with AI in a conversational way and learning how to ask better questions, it feels like I finally have that infinitely patient wizard. Not as a replacement for human connections; rather, a new kind of learning companion that adapts to my particular way of thinking and questioning.

Epistemological Pluralism in the Age of AI

This is where Sherry Turkle's groundbreaking paper "Epistemological Pluralism and the Revaluation of the Concrete" offers valuable insight. Though written decades ago about early computing, her central thesis remains powerfully relevant: there are many valid ways of thinking, learning, and knowing. Many ways of interacting with computational systems.

Turkle observed that traditional computer science education often privileged abstract, rule-based thinking. It simultaneously devalued concrete, contextual approaches. Yet many brilliant minds thrived when allowed to engage with computational systems in ways that honored their preferred learning styles. This was particularly true for those from backgrounds underrepresented in tech.

This perspective connects directly to contemporary learning theory, which increasingly recognizes that effective education must accommodate diverse cognitive approaches rather than imposing a single "correct" methodology. The field of personalized learning and adaptive educational technologies reflects this understanding that different pathways can lead to similar mastery.

I believe we're seeing a similar dynamic play out with AI interaction. The dominant "prompt engineering" paradigm privileges a certain kind of thinking: structured, rule-bound, procedural. But what about those of us who learn differently? What about those who think in metaphors rather than flowcharts? Those who understand through dialogue rather than dictation?

This is why promptagogy matters. It's not about rejecting prompt engineering outright. There's genuine value in those approaches. Rather, it's about expanding the conversation to include diverse ways of learning how to learn from AI.

The Five Principles of Promptagogy

After months of exploration, I've identified five core principles of promptagogy that complement traditional prompt engineering approaches:

1. Observation Alongside Instruction

Study how different people approach the same challenge with AI while learning established techniques.

2. Process As Valuable As Product

The journey of refining questions teaches as much as the final optimized prompt.

3. Diversity of Cognitive Approaches

Different thinking styles produce different questioning strategies, each with unique strengths.

4. Crystallized Thinking

Prompts are artifacts of thought processes that reveal how people conceptualize problems.

5. Collective Learning

We advance faster by sharing our full conversational journeys alongside our successes.

Finding Balance: When to Use Each Approach

Both prompt engineering and promptagogy have their place in AI literacy:

Prompt engineering excels when:

You need consistent, reliable results
You're working within tight constraints or formal systems
Efficiency and optimization are top priorities
You're teaching beginners the fundamentals

Promptagogy shines when:

You're exploring new capabilities or domains
Creative and divergent thinking are valuable
You're building deeper understanding of AI reasoning
Learning styles vary among your team or students

The strongest approach combines both: structured techniques from prompt engineering with the metacognitive awareness and adaptability of promptagogy.

Promptagogy in Practice: A Tool for Collective Learning

This philosophy of learning through questioning inspired me to build promptagogy.com. It's a platform where people can share their prompt journeys, not just their "perfect prompts."

The app allows you to:

Capture a series of prompts from your conversations with ChatGPT or Claude
Annotate each prompt with your thinking, intentions, and reflections
Share your journey with others through a unique URL
Learn from how others evolved their questioning strategies

Unlike platforms that focus solely on sharing optimized prompts, promptagogy.com celebrates the process. The messy middle. The place where real learning happens.

The Process Behind This Article

This piece demonstrates promptagogy in practice. While approximately 80% represents my original thinking, writing, and personal experiences, AI tools helped refine phrasing and generate illustrative examples that comprise roughly 20% of the final text.

In developing this article, I employed several AI-assisted approaches that embody the promptagogy principles I'm advocating. For the editorial review process, I provided AI with the context that it was a viral and award-winning tech writer, then asked for specific feedback on catchiness and readability with my intended academic and practitioner audience in mind.

The term "promptagogy" itself emerged from an AI brainstorming session after exploring many potential names that were either already in use or lacked clarity. I also utilized AI to refine the article's structure, providing principles of readability concerning paragraph length and sentence variation.

The example prompt pairs that illustrate promptagogy in action were generated through my own iterative process with ChatGPT, which itself demonstrates promptagogy principles. I started with a general request:

"Give me 5 pairs of sequential prompts I might ask you, showing an evolution of my questioning style to learn about a topic. Let the first prompt be general or non-specific, and then in the second prompt show refinement that could reasonably have progressed based on the response from an AI."

The initial results were too complicated, technical, and heavily skewed toward software development topics. Seeing this, I refined my own prompt with more specific guidance:

"Make the second/refined prompts simpler. Make them less related to development and more accessible to a general audience. Start the second prompt with something like 'That's too much information' or 'I don't understand' or 'That's not what I wanted'. Give me 5 more pairs."

This process of refining my own prompt to get more useful examples perfectly illustrates the promptagogy approach in practice. I observed the initial results, identified what wasn't working, and evolved my questioning strategy to produce the examples you see in this article.

The core concepts, philosophical framework, and personal anecdotes are entirely my own. This collaborative approach embodies the very principles I'm advocating: an iterative questioning process where human creativity and technological capabilities complement each other. The evolution of this article through conversation with AI systems mirrors the learning journeys that promptagogy aims to document and share.

The Meta-Learning Revolution

The core insight of promptagogy isn't just about learning how to use AI yourself. It's about learning how to learn from how others use AI. It's built on the conviction that seeing how someone else's questions evolved is often more instructive than simply copying their final "perfect" prompt.

By observing the evolution of someone else's dialogue with AI, we gain insight not just into effective prompting, but into different ways of thinking and problem-solving. We learn to recognize patterns of questioning that might never have occurred to us. We expand our own cognitive toolkit by borrowing approaches from others with different thinking styles.

This meta-learning approach represents a fundamental shift. Rather than treating AI interaction as a skill to be mastered through rote techniques, promptagogy treats it as a mirror reflecting our cognitive processes. It becomes a tool for understanding not just the subject matter, but our own approaches to learning itself.

Promptagogy for Different Fields

For Educators

Promptagogy offers a new lens for teaching critical thinking. Instead of presenting AI as a tool to be optimally used, present it as a conversation partner to be explored. Have students compare their questioning strategies and reflect on the differences.

Educational researchers might implement promptagogy by creating "prompt journals" where students document their AI interactions and reflect on how their questioning strategies evolve over a semester. This approach helps develop metacognitive awareness while also providing rich data on diverse learning approaches. Teachers could use these insights to develop more inclusive instructional methods that accommodate various cognitive styles.

For Developers

Beyond crafting the perfect query for code generation, promptagogy encourages studying how different developers approach the same programming challenge through AI conversation. The evolution of a question often reveals more about software design thinking than the final prompt.

In software development teams, promptagogy could manifest as shared AI conversation logs where developers document how they refined their queries to solve specific programming problems. These logs become valuable learning resources, revealing diverse problem-solving approaches and helping new team members understand both the technical solutions and the thought processes behind them. Code reviews could expand to include "prompt reviews" that examine how AI assistance was leveraged.

For Researchers

Academic use of AI often focuses on getting precise, citeable outputs. Promptagogy suggests that the research process itself, the evolution of questions, may be as valuable as the final results.

Researchers implementing promptagogy might create collaborative databases of AI conversations organized by research domain, allowing scholars to study not just what answers were generated but how different researchers framed their inquiries. This meta-research approach could identify discipline-specific questioning patterns and reveal blind spots in conventional research methodologies. For interdisciplinary projects, promptagogy offers a framework for bridging different academic languages and approaches.

Join the Promptagogy Movement

What I've shared here is just my current understanding of promptagogy: the practice of metacognitive learning through AI interaction that prioritizes the observation and documentation of diverse questioning pathways, treating prompts as windows into human thought processes rather than merely as tools for generating specific outputs. I believe this concept deserves broader exploration and contribution.

Promptagogy isn't just about getting better answers from AI, though that's certainly a benefit. It's about becoming more thoughtful questioners. More reflective learners. More aware of our own thought processes. In a world increasingly mediated by AI systems, these meta-skills may prove more valuable than any specific prompt technique.

Promptagogy.com is currently in development. Join the waitlist to be among the first to document and share your prompt journeys: Promptagogy.com Waitlist

How do you learn from AI interactions? Do you follow templates, or develop your own questioning style? Have you ever studied how others interact with AI systems? Share your experiences in the comments below.

Web Scraping Unwrapped: From Python Beginner to Data Harvester

Nicholas Bennett — Sat, 19 Apr 2025 02:20:42 +0000

Ways to Use This Guide:

Read along. Reading Time: ~25 minutes

Execute the code in the Google Colab interactive notebook. You can see the entire article there and run most of the code. You can create a free account with a Google account.

Clone the Github repository and run it on your own computer.

Introduction

Since my post about Uber scraping has gotten an unexpected windfall of attention, I wanted to share what I know about web scraping for a general audience. Whether you're just getting started with Python and want to try this out, or you've heard about web scraping and are curious about what's involved, read on.

Web scraping is a set of programmatic techniques used to extract structured data from the unstructured data of the web. Structured as in it's in a form I want to use, like:

{'title': 'Google', 'url': 'https://www.google.com', 'font_family': 'Roboto, Arial, sans-serif'}

Not unstructured like:

<html><head><title>Google</title>...</body></html>

This article is by no means a canonical description of web scraping, it's just what I know of it. I know that, as a beginner, receiving a deluge of information can be overwhelming for me and I then face a paradox of choice.

If you're a beginner, I suggest that it's more helpful to start with someone else's journey and work from there.

At the end I'll supply some resources that can help you keep going on your own.

Introduction
My Background
How I Scrape
- Python: requests
- Python: requests + json/csv/pandas
- Python: requests + Beautiful Soup
- Python with a Headless Browser: Selenium
Anti-Scraping Technology
- IP Address
- User Agent Filtering / HTTP Headers
- CAPTCHAs / CloudFlare
- Honeypots
- Behavior Analysis
The Ethics of Web Scraping
- Terms of Service: The Digital Fine Print
- Rate Limiting: Don't Be a Resource Hog
- The Data Privacy Angle
- Robots.txt: The Digital "No Trespassing" Sign
- The "Be Reasonable" Test
- Giving Back to the Community
Resources
Conclusion

My Background

I first learned about web scraping in 2009, when I saw some data on a web site that I wanted in a different format. I didn't like their UI and I had an idea in mind for a better interface for me.

Specifically, it was movie data from IMDB, and I wanted to visualize connections between cast and movie titles. Based on actors that I liked I wanted to get recommendations on other movies I'd probably like too.

I was art coding at the time in Processing (a subset of Java). I was making replicas of visual effects from the demoscene. I'd previously used C, x86 assembler, and Pascal/Basic/C#, but none of them gave me the ease of use for web scraping that I wanted.

The best candidates for web scraping at the time were Ruby and Python, and I decided to learn Python because my best friend knew it and I'd have a friendly person to ask questions.

KEY for learning a new language: find a kind mentor

My pursuit of learning web scraping took me to work in a research lab focused on Social Network Analysis where I co-authored two published research papers. This led me on a road of learning Python where I eventually became a Python Developer at a hosting company, and subsequently have used it in every position since. I still web scrape for my personal projects.

How I Scrape

This is how I scrape, from easiest to hardest.

Python: requests

If the information I'm trying to get is available as a GET/POST HTTP request, or if it's in a web page that's not generated by JavaScript, I can use a plain HTTP request using the requests library.

There are other ways to do this, including using the standard library that comes with the Python distribution, but the requests library for Python is built to be simple.

I prefer simple.

This Python code will get me the basic content of https://www.google.com:

import requests

url = "https://www.google.com/"
r = requests.get(url)  # This is where we actually make the HTTP request to fetch the URL
if r.status_code==200:  # HTTP 200 means our request was successful
  html = r.content  # The raw HTML content of the page is stored in r.content
  print(f"Retrieved {url}")  # Just confirming we got the page successfully
else:
  # If we didn't get a 200, something went wrong - maybe the site is down or blocked us
  print(f"Couldn't get {url}, HTTP status code was {r.status_code}")

If you're confused about HTTP codes, they're numbers that a web server sends back to indicate what it thought about the request. Here's a complete index of different codes the web server might send back, and what they mean.

Even if you can get away with using using requests to get the URL, you're going to need to do some parsing of what you get back.

Python: requests + json/csv/pandas

If there's a way to get the data I want from an API rather than the web page itself, and the API returns CSV or JSON (usually), then I can use the built-in json or csv libraries to process the results. If I want to be fancy and do some data processing afterwarsd, I might use pandas.

Here's an example of where I could use csv. Let's say I wanted to know what the low temperature was in Chicago on January 1, 2025. I can use the NOAA NCEI Global Surface Summary of the Day API to find the URL for the weather station at O'Hare Airport and data from 2025, then request the measurements as CSV and parse them to get my answer:

import requests
import csv

# This URL points to weather data for Chicago's O'Hare Airport for 2025
# NOAA provides this data in a nicely structured CSV format - perfect for scraping!
url = 'https://www.ncei.noaa.gov/data/global-summary-of-the-day/access/2025/72530094846.csv'

r = requests.get(url)  # Make the HTTP request for the CSV data
if r.status_code==200:  # Check if we got a successful response
  content = r.content.decode('utf-8')  # Convert the raw bytes to a UTF-8 string
  csv_reader = csv.DictReader(content.splitlines())  # Parse the CSV text into a dict reader
  data = list(csv_reader)  # Convert the reader to a list of dictionaries, one per row
  first_day = data[0]  # Grab just the first day's data (Jan 1, 2025)
  # Print the results in a nice readable format
  print(f"The low temperature on {first_day['DATE']} was {first_day['MIN']} F at {first_day['NAME']}.")
# The low temperature on 2025-01-01 was   27.0 F at CHICAGO OHARE INTERNATIONAL AIRPORT, IL US.

Here's an example of where I could use json. Let's say I want to use the Open Library APIs to get the bibliography of Thich Nhat Hanh, a prolific Buddhist monk whose writings have been immensely influential in my life. You can query their Authors API to search by name, then use the unique identifier for a particular author to get their list of works.

One convenience here is that because data is often returned as JSON from an API, you don't need to separately import json to decode JSON data. You can use the .json() method on the requests object after it returns.

import requests

# This URL searches for authors with the name "thich nhat hanh" in the Open Library API
search_url = 'https://openlibrary.org/search/authors.json?q=thich%20nhat%20hanh'
r = requests.get(search_url)  # Make the HTTP request to the API
if r.status_code==200:  # If the request was successful
  results = r.json()  # Parse the JSON response directly into a Python dictionary
  for result in results['docs']:  # Loop through each author result
    # Print a neatly formatted summary of each author and their work count
    print(f"Key: {result['key']} Author: {result['name']} ({result['work_count']} works on record)")

# Now we'll use the first result (which has the most works) to find their books
author_record = results['docs'][0]  # Grab the first author record (most works)
# Construct a new URL to get up to 1000 works by this author
works_url = f"https://openlibrary.org/authors/{author_record['key']}/works.json?limit=1000"
r = requests.get(works_url)  # Make another HTTP request for the works
if r.status_code==200:  # If this request was also successful
  results = r.json()  # Parse this JSON response
  works = results['entries']  # Get the list of works
  print(f"Found {len(works)} works for {author_record['name']}.")
  print("Here are the first 10 titles:")
  # Loop through the first 10 works and print their titles
  for work in works[:10]:
    print(work['title'])

Let's revisit the weather downloading example to show the power of pandas and its data analysis abilities. Let's say we want to find the day in 2025 so far that was the coldest in Chicago:

import requests
import pandas as pd

# Same Chicago weather data URL as before
url = 'https://www.ncei.noaa.gov/data/global-summary-of-the-day/access/2025/72530094846.csv'
r = requests.get(url)  # Make the HTTP request
if r.status_code==200:  # If successful
  # Convert the response to a CSV string and use pandas to parse it directly
  # This is a slick one-liner that replaces multiple steps from the previous example
  df = pd.read_csv(pd.io.common.StringIO(r.content.decode('utf-8')))

  # Here's where pandas shines - we can easily find the coldest day
  # idxmin() gives us the index of the minimum value in the 'MIN' column
  coldest_day = df.loc[df['MIN'].idxmin()]

  # Print out our finding in a readable format
  print(f"The coldest day of 2025 so far was {coldest_day['DATE']} with {coldest_day['MIN']}°F at {coldest_day['NAME']}.")
# The coldest day of 2025 so far was 2025-01-20 with -5.0°F at CHICAGO OHARE INTERNATIONAL AIRPORT, IL US.

Python: requests + Beautiful Soup

Other times, a web page comes back from a direct HTTP GET as just the HTML. Then I can use an HTML/XML parser like Beautiful Soup.

BeautifulSoup4 gives easier access to specific elements in the web page through chains of selectors in the page's Document Object Model (DOM). The DOM is just a tree data structure representing the elements and sub-element relationships. For example, because both head and body are singular sub-elements of the main html element in an HTML file, Then both head and body are considered children of the html element.

Let me show you some code:

import requests
from bs4 import BeautifulSoup

url = "https://lxml.de/"
r = requests.get(url)  # Make the HTTP request for the page
if r.status_code==200:  # If successful
  # Create a BeautifulSoup object from the HTML content
  # This parses the HTML into a navigable structure
  soup = BeautifulSoup(r.content)
  print(f"Retrieved {url}")

  # With BeautifulSoup, we can directly access elements like title
  title = soup.title.text  # Get just the text inside the <title> tag
  body = soup.body.text  # Get all text inside the <body> tag

  # We can also use CSS selectors to find specific elements
  # This finds the first h1 tag with class "title" and extracts its text
  headline = soup.select('h1.title')[0].text  # 'lxml - XML and HTML with Python'
else:
  print(f"Couldn't get {url}, HTTP status code was {r.status_code}")

With BeautifulSoup, you can directly access elements from the convoluted HTML structure with things like soup.title, and you can use CSS Selectors to find almost any content on the page.

Most scraping jobs can be be done with some combination of these. However, as we move towards a more dynamic web with pages often built using client-side JavaScript (common with libraries like React, AngularJS, and Vue.JS), then you need to simulate a browser.

Python with a Headless Browser: Selenium

Using a headless browser is advanced stuff. It's a way of using code to make a browser do work for you, but an actual browser needs to be installed just for your program. This won't work well in a web-based Python environment, so if you want to try this out you would need to set this up on your own computer.

This is useful when a website is generating data at least partially with JavaScript, and either you don't want to or it's too time-consuming to follow up the stream from the JavaScript to tap the original data source.

A browser like this is "headless" because it doesn't give you, the user, a head to talk to it. It's a puppet of the program. This browser is otherwise fully functional, and does the heavy lifting of executing JavaScript, rendering a page, and giving access to interact with the interface. After leading a headless browser do all of this work, your program can siphon off the created HTML to then parse externally (with something like Beautiful Soup).

There are several options here, but because I'm focused on Python in this article I'll stick with what I know best - Selenium.

The other options I've heard good things about are Playwright and Puppeteer which are written in JavaScript.

A common use case for these headless browsers is e-commerce sites, where stock is changing all the time and live code actually builds what you see.

Let's try to find the latest items on sale for men from one of my favorite brands, Uniqlo.

Here's the URL for items on sale as of 4/18/2025: https://www.uniqlo.com/us/en/feature/sale/men?storeId=125789&inventoryCondition=1

They're almost assuredly rendering the page partly with JavaScript. It may be possible to dive under the hood of the JavaScript or network requests to find the API request directly, but if we're trying to get data without too much work we can use a headless browser.

Here's how we'd get the raw HTML of the site:

from selenium import webdriver
from bs4 import BeautifulSoup

# A URL that probably uses JavaScript to load its content
url = "https://www.uniqlo.com/us/en/feature/sale/men?storeId=125789&inventoryCondition=1"

# Initialize a Chrome browser instance that Selenium will control
driver = webdriver.Chrome()

# Tell the browser to load our URL
driver.get(url)

# Wait 5 seconds for the page to fully load
# This gives JavaScript time to render the page content
driver.implicitly_wait(5)

# Get the fully rendered HTML (after JavaScript execution)
# and parse it with BeautifulSoup
soup = BeautifulSoup(driver.page_source)

# Always close the browser when we're done to free up resources
driver.quit()

# Print the page title as a simple test
print(soup.title.text)
# Sale | Clothing & Accessories for Men | UNIQLO US

In this article I'm not going to get into specifics of setting up Selenium, but to the casual reader I will say it's a bit cumbersome. You have to install a platform-specific chromedriver that lets Selenium use it as its pretend web browser.

For Mac this is relatively easy using Homebrew: brew install chromedriver before you execute this script.

If you're on Windows, here's a tutorial.

If you're on Linux, I don't need to explain anything to you (just kidding, here you go).

Anti-Scraping Technology

There are all manner of techniques that web sites use now to prevent scraping. This is by no means a canonical list, this is just what I know:

IP Address

From a numeric 12.234.211.6 IP address, it's possible to determine where it's located - even without being law enforcement. Everything on the Internet is just a series of hops away, through DNS and routers, and common command-line tools can suggest where something is located based on the names of preceding servers.

There are far more sophisticated tools and libraries out there for recognizing IP addresses besides eyeballing, and it can be easy for a web server to tell if the IP address of your request is coming from a home address (expected) or a datacenter address (less expected).

If you're running your scraper from the cloud (such as AWS / Azure / Google Cloud) and your scraper is getting blocked by a website, your best bet is to use rotating proxies. A proxy will be an address your traffic appears to come from, while your program is actually asking the proxy to make the request for you and send it back results.

User Agent Filtering / HTTP Headers

Every web browser has a User Agent string that it sends to a web site (you can check yours here). This identifies the browser and, to a certain extent, the operating system and device you're using (read more here).

The requests library and any of the headless browsers all identify themselves with implicit User Agents which are easy to detect.

import requests

url = "https://httpbin.io/user-agent"
r = requests.get(url)  # Make a request to a service that returns our User-Agent
print(r.json()['user-agent'])  # Extract and print the User-Agent from the JSON response
# This will show something like 'python-requests/2.31.0'
# Sites can easily detect this isn't a real browser!

It's easy to tell this is coming from requests, not a real browser. Now let's look at how Selenium appears:

from selenium import webdriver
from bs4 import BeautifulSoup
import json

url = "https://httpbin.io/user-agent"  # Service that shows our User-Agent
driver = webdriver.Chrome()  # Start up a Chrome browser
driver.get(url)  # Load the URL
html = driver.page_source  # Get the rendered page source
driver.quit()  # Close the browser

# Parse the HTML and extract the User-Agent from the JSON response
soup = BeautifulSoup(html)
print(json.loads(soup.text)['user-agent'])
# This will show something like 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36...'
# Much more like a real browser!

There are other HTTP headers that are sent by browser but typically not by scrapers, so to make a website think your scraper is a browser it's best to replicate all of the HTTP headers from a real browser (here's one place you can check).

CAPTCHAs / CloudFlare

The human-checking applets we've all come to accept, but often been frustrated by, are in place to defeat web scrapers.

These are quite effective, and defeating these typically requires some commercial scraping service or some underground CAPTCHA-defeating program. Machine Learning is helpful here.

Honeypots

For a real person using an Internet device, you won't be able to click on invisible content. But for a scraper, it sees ALL and might inadvertently access invisible elements. If you're using a headless browser, it's important to ignore HTML tags with styles like display: none or visibility: hidden.

Behavior Analysis

Most human users of a website take a few seconds to look at a page before clicking on a link, and it's an irregular amount of time per page. They also probably don't click on every single item on a web page.

If your scraper just requests every page as quickly as possible without delays, or with identical delays like one second each, then it would be easy for the web server to tell that the scraper requests are not coming from a real person.

To bypass this restriction, it can be important to insert semi-random wait times between requests, and also split up requests across multiple IP addresses to better simulate a real person's browsing activity.

The Ethics of Web Scraping

When I first started scraping websites back in 2009, the ethical considerations barely crossed my mind. I just wanted my movie data! But over the years, I've developed a more nuanced view of what it means to be a responsible scraper. Let me share what I've learned about navigating these murky waters.

Terms of Service: The Digital Fine Print

Most websites have a Terms of Service (ToS) document that explicitly mentions scraping. Some forbid it outright, while others place limitations on how you can access their data. Before you start scraping a site, take a few minutes to actually read this document. It's usually linked at the bottom of the homepage, and while it's tempting to ignore it like we all ignore software EULAs, this one matters. (check out this ToS decoder)

For example, LinkedIn's ToS explicitly prohibits "scraping, spidering, crawling or other technology to access data on the site." Meanwhile, sites like Wikipedia are generally more permissive, though they still have guidelines about rate limiting (more on that in a bit).

Remember that just because you can scrape a site doesn't mean you should. Ask yourself: Would the site owner reasonably expect this kind of access to their data? Am I potentially harming their business model?

Rate Limiting: Don't Be a Resource Hog

Imagine if you ran a website and someone hammered your server with 100 requests per second. Your site would slow down or crash, affecting all your legitimate users. That's why I almost always implement rate limiting in my real scrapers.

import requests
import time

# List of URLs we want to scrape
urls = ["https://example.com/page1", "https://example.com/page2"]
contents = []

for url in urls:
    # Request the URL
    r = requests.get(url)

    # Store the content in an array for later processing    
    contents.append(r.content)

    # Wait between requests to avoid hammering the server
    # Being a good digital citizen means not overloading websites
    time.sleep(1)  # 1 second delay between requests

This simple 1-second delay can make the difference between being a good digital citizen and inadvertently launching a denial-of-service attack. For sites that don't explicitly state rate limits, I start conservative (1 request every few seconds) and only speed up if I'm confident it won't impact the site.

Some larger sites publish their rate limits in their developer documentation or robots.txt file. Always respect these limits. If they publish the rate limits, you can bet they'll have consequences if you violate the limits.

The Data Privacy Angle

In the age of GDPR, CCPA, and other privacy regulations, we need to consider what kind of data we're scraping. Public facts are one thing, but personal information is another matter entirely.

Ask yourself: If someone collected this data about me, would I be uncomfortable? If yes, reconsider your approach.

Robots.txt: The Digital "No Trespassing" Sign

Most websites have a file called robots.txt that tells automated crawlers which parts of the site they can access. While it's primarily intended for search engines, it's also a strong indicator of what the site owner considers off-limits for automated access.

Here's how to check and respect robots.txt:

import requests
from urllib.robotparser import RobotFileParser

def is_allowed(url, user_agent="MyScraperBot"):
    # Create a robot parser object
    rp = RobotFileParser()

    # Construct the URL to the site's robots.txt file
    # This splits the URL and rebuilds just the domain part + /robots.txt
    robots_url = url.split('/')[0] + '//' + url.split('/')[2] + '/robots.txt'

    # Set the URL and read the robots.txt file
    rp.set_url(robots_url)
    rp.read()

    # Check if our user agent is allowed to access this URL
    return rp.can_fetch(user_agent, url)

url = "https://example.com/interesting-data"
if is_allowed(url):
    # We're allowed to scrape this URL, so proceed
    r = requests.get(url)
else:
    # We're not allowed, so be respectful and don't scrape
    print("This URL is disallowed by robots.txt")

The "Be Reasonable" Test

When all else fails, I apply what I call the "be reasonable" test. If I were running the site I'm about to scrape, what would I consider reasonable use of my data?

Would I be okay with someone:

Making a full copy of my site? (Probably not)
Extracting specific structured data for personal use? (Maybe)
Building a competing service with my data? (Definitely not)

If you're still unsure, reach out to the site owner directly. You'd be surprised how many are willing to help if you explain your project and approach them respectfully.

Giving Back to the Community

If you benefit from scraping a website, consider how you might give back. Maybe you can share insights from your analysis, contribute to open-source projects that help the site, or even offer to help improve their API if they have one.

The web is built on a foundation of sharing information. Let's keep that spirit alive by being ethical scrapers who respect the ecosystem we're drawing from.

Resources

A one-stop shop for web scraping tools for a variety of languages is collected in the lorien/awesome-web-scraping Github repository.

Getting started with a development environment with any language on your computer can be one of the hardest things to navigate as a beginner. Here are some ways to make that easier with Python:

Google CoLab - I have the example code of this article hosted there to enable you to execute the code easily. You can set up your own account and create code you can execute in the browser there.
Anaconda Python Distribution - This is a batteries-included Python distribution that includes several editors and a streamlined way to install and manage packages. This is popular in industry, academia, and research. You would need to use something like this if you wanted to use a headless browser.
ChatGPT - Invaluable tool for explaining errors, code concepts, and generating example code.
Claude AI - Same as with ChatGPT, though I've found a little better presentation of code examples with Claude.
CodeMentor - If you want a real live human to help you with programming, CodeMentor is a service that connects you with someone you can relate to.
FreeCodeCamp - If you want to avoid spending money to find a mentor, you can find a supportive community of beginner-friendly programmers at FreeCodeCamp.

Conclusion

As you can see, web scraping can be a complex and vast topic. It's a constant cat and mouse game of people building web pages/apps and other people finding ways to scrape that data and functionality. Applications range from liberating data for your own use to getting pricing information from a competitor.

What started for me as a simple desire to get movie data has evolved into a skill I've used throughout my career. The web is essentially the world's largest, messiest database, and scraping is my imperfect tool to make sense of it. As you start your own scraping journey, remember that with great power comes great responsibility. Be ethical, be considerate, and maybe even reach out to site owners when in doubt.

The most rewarding scraping projects are often ones that unlock data for purposes the original creators never imagined. Maybe you'll visualize weather patterns in your hometown, build a better interface for public transit information, or create a personal dashboard of information that matters to you.

What do you think about scraping? If you could scrape a website, which one would it be? I'd love to hear what ideas you have! Drop a comment below or connect with me directly to share your thoughts or questions.

Nicholas Bennett

LinkedIn | Github | Website

Vibe Coding 4,379 Uber Rides Into a Living Map

Nicholas Bennett — Sun, 13 Apr 2025 18:20:03 +0000

Introduction

Have you ever seen a product in your mind's eye but been unsure of how to create it?

Imagine you’re a full-time Uber driver. What if you could watch every mile of your Uber driving history come to life on a map?

In 2023 and 2024 I drove a total of 4379 Uber rides, and I was able to scrape the Uber Drivers page to get this data (see how I did this here). I invested countless hours in this work, exploring Chicago's neighborhoods and meeting so many interesting people, and I didn't want this experience to be relegated to a CSV. I wanted a living visual (built using what I now understand is called vibe coding: creative, exploratory programming powered by AI assistance).

I envisioned an animated map showing all the trips I took and how I moved between them. This was something I could show to my family and friends, explaining to them where I went better than raw data or mere numbers. More importantly, it would help jog my memory of my myriad experiences.

I just didn't have much experience in web animation or GIS, and I balked when I thought of the time investment and frustrations along the way to get to what I wanted.

TL;DR:

I scraped 4379 Uber rides, geocoded the routes, and used AI-assisted vibe coding with ChatGPT to build an animated map of every trip — day by day — using Python, Mapbox, and GeoJSON. Here it is!

The Vibe Coding Process

Thankfully, now with the power of code-aware LLMs, it's possible to get mostly functional examples of code based on plain English descriptions and be able to iterate through versions with the LLM. In this article I specifically used ChatGPT, with a combination of their o3-mini-high (more code-correct) and ChatGPT 4o models. My development process was roughly {LLM: initial prompt} -> {manual tweaking} -> {LLM: code tweaking, feature adding, debugging} -> {manual tweaking} ....

In this article I'll show you the power of using LLMs and vibe coding in conjunction with language fluency (in this project, Python and JavaScript) to bring an idea to life. When facing domains I didn't have a lot of experience in (GIS and web animation), and being a learner more from examples than reference, it was extremely helpful to get examples of semi-working code from the LLM and then be able to learn concepts backwards.

Even this article is somewhat vibe-coded. I gave ChatGPT a prompt that it's an award-winning and viral writing coach and asked it for a narrative structure and a number of catchy titles. It broadly outlined the article and I iterated with it to find the title that was most "me". The prose in the body of this article is all me, but the skeleton was aided considerably by AI. I used it as well to analyze my language and point out potential readability issues in my language, considering my audience of fellow coders and hiring managers for the kinds of roles I want.

The Challenge

Having the intuitive sense that an animation of all 4379 rides at once would be spaghetti on the map and unintelligible, I realized I'd want to show my Uber driving day-by-day. Since I often drove late into the night for people going out, I had to unconventionally set my "day" to 5:00 AM - 4:59 AM the next day.

I also knew that the addresses in my data were non-specific:

In [1]: import pandas as pd
In [2]: rides = pd.read_csv('../../data/rides.csv')
In [3]: rides['pickup_address'].head()
Out[3]: 
0    N Ashland Ave, Chicago, IL 60614-1105, US
1         N Lincoln Ave, Chicago, IL 60613, US
2    N Clifton Ave, Chicago, IL 60657-2224, US
3    W Belmont Ave, Chicago, IL 60657-4511, US
4     N Whipple St, Chicago, IL 60647-3821, US
Name: pickup_address, dtype: object

I would need a robust geocoding service that could handle these locations. The geocoding would result in (latitude, longitude) coordinates for each pickup_address and dropoff_address.

I also didn't want to just draw lines between these coordinates, which would ignore real streets. So I needed a routing API to give me street-based paths based on two lat/long coordinates.

From these routes, I would then want to display them on a live map on the web. I knew I needed a third API to show a map overlaid with markers and paths. I just didn't know what format of data this would require.

Solution Exploration with ChatGPT

Through talking to ChatGPT, I was able to learn a lot more about GIS, specifically that these paths would be called geometries and that I could store these as GeoJSON files which would then be usable in a live map.

Here's a prompt I used to start to get familiar with the technology out there:

I have a list of locations with zip code and sometimes a street name, but no specific house numbers. They are time stamped. I want to build an animation of movement between these points in a car, using roads that one would take between those locations using routing software like Google Maps.
How could I do this?
Alternately, how could I generate the GPX file so I could use it with third party software?

I had learned about GPX files from some map animation apps I'd explored, which had paid tiers. I thought if I couldn't find a way to do it myself, then I could at least get the files and feed them to the external service. I found out in this line of questioning, however, that I wouldn't need an external service and that GeoJSON would be more suitable for my task.

I then asked it these clarifying questions:

Explain the difference between the Leaflet and Mapbox GL JS options
I definitely want to use Mapbox GL JS. I might break up animation on a day by day basis in which case there’d be a maximum of 50 locations per day. But I might also want to animate the entire data set which is then 4300 locations

It gave me a full breakdown of how I could do this with all three APIs I was looking for coming from Mapbox. Now I was concerned about cost, because I'm on a very limited budget for this personal project:

What is the cost of the Mapbox directions API for route finding? Give me recommendations with price comparisons on getting road routing information given my data set is 4300 points.

Thankfully, it told me that Mapbox has a free tier which allows for 100K queries in a month. It showed me a few other services, but Mapbox was clearly the winner with a cohesive set of APIs and beautiful map tiles. It offered:

What I’d Do in Your Shoes
Start with Mapbox: You’re under the 100,000 req/month and using Mapbox GL JS.
Throttle batch calls and cache results as GeoJSON to avoid repeated API hits.
If you exceed quota later, move to ORS or self-host OSRM.
Want a script to:
Cache route requests to disk as GeoJSON or GPX?
Automatically throttle and deduplicate?
Let me know — I can write that out for you.

I told it "Yes". Here we go!

Step 1: Geocoding and Routing with Mapbox

ChatGPT provided me an overview of the work it was going to do, enlightening me to details I hadn't even thought to ask, and it gave me some clarifying questions so it could work better:

Awesome. Here's what I’ll provide:

A Python script that:

Reads your list of 4300+ locations.

Sends route requests to Mapbox Directions API (or OpenRouteService).

Throttles requests to stay within rate limits.

Caches all responses locally (as GeoJSON) to avoid repeat API calls.

Optionally writes a combined GPX or GeoJSON route file for animation or import.

Modular design so you can easily switch routing backends (Mapbox or ORS).

Questions Before I Start

To make this precise for you, please confirm the following:

Format of your input:

Is it a CSV file?

What columns does it have? (e.g., timestamp, zip, street, lat, lon, etc.)

Do you already have latitude/longitude, or do I need to geocode from zip/street?

Which routing backend would you like to start with?

Mapbox Directions API

OpenRouteService (ORS)

(We can swap easily later.)

Do you want the output to be:

One long GPX file?

One file per day?

Both?

Once I have those answers, I’ll write your script.

I told it:

Input is a CSV file. It has columns address, zip, timestamp. You need to geocode from address ideally or fall back to zip if that doesn’t work or if results are ambiguous. Start with Mapbox Directions API. Both outputs.

It asked for a sample of my data, so I uploaded my entire 796 KB rides.csv file. I had to provide clarifying prompts subsequently for some pertinent details:

The start address was in the pickup_address column and the end address was in the dropoff_address column.
I wanted a route between the end address of one ride to the start address of the next ride.
The ZIP was actually in the address fields already so it could ignore the zip column.
I didn't need GPX output after all and it could just generate GeoJSON.
I didn't need a combined GeoJSON file of all routes, I just wanted them broken out by day.
I wanted to filter out any rides where the geocoded address was more than 100 miles from the center of Chicago (some addresses were so generic that they geocoded to Montana).
It kept trying to run the Python code in chat session (!!) so I told it I wanted a script I could run locally.
That a day would end at 5 AM instead of midnight.
I wanted to add metadata to each feature (an individual route between points within the resulting GeoJSON file) from rides.csv, particularly whether it was an actual ride or a transition between rides.
I wanted debugging information displayed to the console so I could see progress of the script.
I wanted it to bake in retries of the API endpoints with waits in case there were network problems.

Based on the example it provided of adding metadata to the GeoJSON features, I added in my own code to add additional metadata to ride features from rides.csv including timestamp, ride_start, ride_end, ride_type, pickup_address, dropoff_address, base_pay, surge, tip, earnings, duration, and distance.

I created a Mapbox account, created a private key for doing these requests to the Mapbox Geocoding v5 API and Mapbox Directions v5 API, stored that locally in a .env file, and loaded that with the python-dotenv library.

Amazingly, this script worked on first run!

import pandas as pd
import hashlib
import requests
import time
import json
import os
from pathlib import Path
from collections import defaultdict
import math
from dotenv import load_dotenv

load_dotenv()

MAPBOX_TOKEN = os.getenv('MAPBOX_TOKEN')
INPUT_CSV = "../../data/rides.csv"
CACHE_FILE = "cache.json"
OUTPUT_DIR = "output/geojson"
GEOCODE_BASE = "https://api.mapbox.com/geocoding/v5/mapbox.places/"
DIRECTIONS_BASE = "https://api.mapbox.com/directions/v5/mapbox/driving/"
MAX_RETRIES = 3
RETRY_DELAY = 2

CHICAGO_LAT = 41.8781
CHICAGO_LON = -87.6298
MAX_MILES = 100

# Initialize cache
Path(OUTPUT_DIR).mkdir(parents=True, exist_ok=True)
if os.path.exists(CACHE_FILE):
    with open(CACHE_FILE) as f:
        cache = json.load(f)
else:
    cache = {}

def save_cache():
    with open(CACHE_FILE, "w") as f:
        json.dump(cache, f)

def cache_key(*args):
    return hashlib.md5("::".join(args).encode()).hexdigest()

def haversine(lat1, lon1, lat2, lon2):
    R = 3958.8  # miles
    phi1, phi2 = math.radians(lat1), math.radians(lat2)
    d_phi = phi2 - phi1
    d_lambda = math.radians(lon2 - lon1)
    a = math.sin(d_phi / 2) ** 2 + math.cos(phi1) * math.cos(phi2) * math.sin(d_lambda / 2) ** 2
    return 2 * R * math.asin(math.sqrt(a))

def is_valid_point(lat, lon):
    return haversine(lat, lon, CHICAGO_LAT, CHICAGO_LON) <= MAX_MILES

def safe_request(url):
    for attempt in range(MAX_RETRIES):
        try:
            r = requests.get(url)
            if r.status_code == 200:
                return r.json()
            print(f"[HTTP {r.status_code}] {url}")
        except Exception as e:
            print(f"[Request Failed] {e}")
        time.sleep(RETRY_DELAY)
    return None

def geocode(address):
    print(f"[GEOCODE] {address}")
    key = cache_key("geocode", address)
    if key in cache:
        return cache[key]
    url = f"{GEOCODE_BASE}{requests.utils.quote(address)}.json?access_token={MAPBOX_TOKEN}&limit=1"
    data = safe_request(url)
    coords = None
    if data and data.get("features"):
        coords = data["features"][0]["geometry"]["coordinates"]
        # Filter out-of-bounds geocoded results
        if not is_valid_point(coords[1], coords[0]):
            print(f"[SKIPPED] Geocoded point {coords} too far from Chicago")
            coords = None
    cache[key] = coords
    return coords

def route_between(c1, c2):
    if not is_valid_point(c1[1], c1[0]) or not is_valid_point(c2[1], c2[0]):
        print(f"[SKIPPED] One or both points out of range: {c1}, {c2}")
        return None

    print(f"[ROUTE] {c1} -> {c2}")
    key = cache_key("route", str(c1), str(c2))
    if key in cache:
        return cache[key]

    coords_str = f"{c1[0]},{c1[1]};{c2[0]},{c2[1]}"
    url = f"{DIRECTIONS_BASE}{coords_str}?access_token={MAPBOX_TOKEN}&geometries=geojson"
    data = safe_request(url)

    geometry = None
    if data and data.get("routes"):
        route = data["routes"][0]["geometry"]
        coords = route.get("coordinates", [])
        coords = [pt for pt in coords if is_valid_point(pt[1], pt[0])]
        if coords:
            route["coordinates"] = coords
            geometry = route
        else:
            print("[SKIPPED] All route points filtered out as outliers")

    cache[key] = geometry
    return geometry

def save_geojson(day, features):
    geojson = {
        "type": "FeatureCollection",
        "features": features
    }
    out_path = os.path.join(OUTPUT_DIR, f"{day}.geojson")
    with open(out_path, "w") as f:
        json.dump(geojson, f)
    print(f"[SAVED] {out_path}")

rides_df = pd.read_csv(INPUT_CSV)
rides_df["ride_start"] = pd.to_datetime(rides_df["ride_start"])
rides_df = rides_df.sort_values("ride_start")

# Geocode pickup and dropoff
rides_df["pickup_coords"] = rides_df["pickup_address"].apply(geocode)
rides_df["dropoff_coords"] = rides_df["dropoff_address"].apply(geocode)
rides_df = rides_df[rides_df["pickup_coords"].notnull() & rides_df["dropoff_coords"].notnull()]

routes_by_day = defaultdict(list)

for i in range(len(rides_df)):
    ride = rides_df.iloc[i]
    ride_time = ride["ride_start"]
    if ride_time.time() < pd.to_datetime("05:00").time():
        day = (ride_time - pd.Timedelta(days=1)).date()
    else:
        day = ride_time.date()

    # 1. pickup -> dropoff
    geometry1 = route_between(ride["pickup_coords"], ride["dropoff_coords"])
    if geometry1:
        routes_by_day[day].append({
            "type": "Feature",
            "geometry": geometry1,
            "properties": {
                "timestamp": ride["ride_start"].isoformat(),
                "type": "ride",
                "rideType": ride["ride_type"],
                "pickupAddress": ride["pickup_address"],
                "dropoffAddress": ride["dropoff_address"],
                "basePay": f"{float(ride['base_pay']):.2f}",
                "surge": f"{float(ride['surge']):.2f}",
                "tip": f"{float(ride['tip']):.2f}",
                "earnings": f"{float(ride['earnings']):.2f}",
                "duration": f"{int(ride['duration'])}",
                "distance": f"{float(ride['distance']):.2f}",
                "rideStartTime": ride['ride_start'].strftime("%B %d, %Y at %I:%M %p"),
                "rideEndTime": pd.to_datetime(ride['ride_end']).strftime("%B %d, %Y at %I:%M %p")
            }
        })

    # 2. dropoff -> next pickup (if available)
    if i < len(rides_df) - 1:
        next_ride = rides_df.iloc[i + 1]
        geometry2 = route_between(ride["dropoff_coords"], next_ride["pickup_coords"])
        if geometry2:
            routes_by_day[day].append({
                "type": "Feature",
                "geometry": geometry2,
                "properties": {
                    "timestamp": next_ride["ride_start"].isoformat(),
                    "type": "transition"
                }
            })

for day, features in routes_by_day.items():
    save_geojson(str(day), features)

save_cache()
print("✅ Done generating cleaned route files.")

Voila! Now I had 339 individual daily GeoJSON files in my output/geojson folder.

Step 2: Building Interactive Map with Mapbox GL JS and Turf.js

All of these files won't do me much good unless I can visualize them. I don't want to replicate the full set of iterations of prompts I gave ChatGPT to generate this, because this was hundreds of prompts over several days. The steps were a variety of:

I tried AI-assisted tweaks that didn't work out the way I wanted,
went back to previous versions,
sent my manually tweaked code to ChatGPT to update its memory,
pasted in errors from the browser's dev console to debug them,
did considerable additions to the code myself after I learned how it all worked by example.

I also had to write my own script to produce the static array of GeoJSON file paths in the file days.js, since I knew I'd be hosting this on Github Pages and I wasn't sure if the client-side script would be able to access an index listing of the output/geojson file.

Instead, of showing you every prompt I used over all the days, let me offer you a reverse-generated prompt. I gave ChatGPT the final HTML/JS code and asked it:

This is the actual HTML/JS produced, make a prompt that would generate this. Tailor this for use with ChatGPT AI using specifically the o3-mini-high model. Give me the prompt in a format I can easily copy and paste to use in a Markdown document.

The prompt it gave me, which I think describes the project well, is:

Prompt for ChatGPT (o3-mini-high): Route Animation Web App

Build a complete index.html file that displays an animated driving route using Mapbox GL JS. The animation should read from a local GeoJSON file per day and animate a car marker driving the route point-by-point.

🗺️ 1. Map

Use Mapbox GL JS v2.15.

Set center to [-87.65, 41.9] (Chicago).

Use the light map style.

Load and display a route as a dashed, semi-transparent blue line.

📦 2. Data Input

Route data comes from a local GeoJSON file: output/geojson/{DAY}.geojson.

The file contains a FeatureCollection with multiple LineString features.

Each feature has properties.type of either "ride" or "transition", and may include:

pickupAddress, dropoffAddress, basePay, tip, surge, earnings

distance, duration, rideType, rideStartTime, rideEndTime

🚙 3. Animation

Animate a 🚙 emoji marker along the entire route.

Calculate a per-coordinate animation delay:

For "ride" features with duration, use: (duration / 400) * 1000 / number_of_coordinates

Use a fixed default delay of 100ms for all other segments.

On each animation step:

Move the marker to the next coordinate.

Update the trip number.

Show a trip description panel with styled HTML and emojis.

📍 4. Start/End Markers

For each "ride" feature:

Add a 🟢 marker at the start

Add a 🛑 marker at the end

On hover, show a popup with the ride description

📆 5. UI

Include:

A <select> dropdown to choose a day

Start / Pause / Resume buttons

A trip info display panel at the bottom left

Load available days from days.js, which defines: var days = ["2024-01-06", "2024-01-07", ...]

Randomly pick a default day on load and auto-start the animation

💅 6. Style

All CSS should be included in a <style> tag

Use rounded corners, soft shadows, and a clean sans-serif font

Style popup cards to show structured ride info with emoji icons

Route line should use:

A dashed stroke

Blue color (#007cbf)

80% opacity

🔧 Dependencies (no build tools)

Only use:

https://api.mapbox.com/mapbox-gl-js/v2.15.0/mapbox-gl.js

https://cdn.jsdelivr.net/npm/@turf/turf@6/turf.min.js Output must be a standalone index.html file.

I used a public key from the Mapbox API, which it states is built to be shared publically and seen in client-side code. Here's the final rides.html:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <title>Route Animation</title>
    <meta name="viewport" content="initial-scale=1, maximum-scale=1, user-scalable=no" />
    <link href="https://api.mapbox.com/mapbox-gl-js/v2.15.0/mapbox-gl.css" rel="stylesheet" />
    <style>
      body {
        margin: 0;
        padding: 0;
      }
      #map {
        position: absolute;
        top: 0;
        bottom: 0;
        width: 100%;
      }
      #controls {
        position: absolute;
        top: 10px;
        left: 10px;
        z-index: 1;
        background: white;
        padding: 10px;
        border-radius: 4px;
        box-shadow: 0 1px 3px rgba(0, 0, 0, 0.3);
        font-family: sans-serif;
      }
      #descriptionDisplay {
        position: absolute;
        bottom: 10px;
        left: 10px;
        z-index: 1;
        background: white;
        padding: 10px;
        border-radius: 4px;
        box-shadow: 0 1px 3px rgba(0, 0, 0, 0.3);
        font-family: sans-serif;        
      }
      label,
      select {
        display: block;
        margin-bottom: 6px;
        width: 100%;
      }
      #status {
        margin-top: 10px;
        font-weight: bold;
        font-size: 14px;
      }
      .mapboxgl-popup {
        z-index: 100 !important;
      }
      .popup-content {
        font-family: "Helvetica Neue", Arial, sans-serif;
        padding: 10px;
        border: 1px solid #ddd;
        background-color: #fff;
        box-shadow: 0 1px 3px rgba(0,0,0,0.3);
        border-radius: 4px;
      }
      .popup-content h3 {
        margin-top: 0;
        color: #333;
      }
      .popup-content p {
        margin: 4px 0;
        font-size: 14px;
        line-height: 1.3;
      }
      .popup-content hr {
        border: none;
        border-top: 1px solid #eee;
        margin: 8px 0;
      }
      .popup-content strong {
        color: #555;
      }
      .popup-icon {
        margin-right: 4px;
      }
    </style>
    <script src="https://api.mapbox.com/mapbox-gl-js/v2.15.0/mapbox-gl.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/@turf/turf@6/turf.min.js"></script>
  </head>
  <body>
    <div id="map"></div>
    <div id="controls">
      <label for="daySelect">Select Day</label>
      <select id="daySelect"></select>

      <button id="startButton">Start</button>
      <button id="pauseButton">Pause</button>
      <button id="resumeButton">Resume</button>

      <div id="status">
        Trip <span id="rideNumberDisplay">–</span>
      </div>
    </div>
    <div id="descriptionDisplay"></div>

    <!-- days.js should define a variable "days", for example: -->
    <!-- var days = ["2024-01-06", "2024-01-07"]; -->

    <script src="days.js"></script>
    <script>
      // Set Mapbox access token and initialize the map.
      mapboxgl.accessToken =
        "pk.eyJ1IjoibmljaG9sYXM3NzciLCJhIjoiY202NW40cHpnMXJmODJpcHlvZHVjZDA5ZiJ9.HoZYIgpxoOkZEM7Fb9_Lsw";
      const map = new mapboxgl.Map({
        container: "map",
        style: "mapbox://styles/mapbox/light-v11",
        center: [-87.65, 41.9],
        zoom: 11
      });

      // Animation variables.
      let marker = null;
      let animationCoords = [];
      let animationIndex = 0;
      let animationFrameId = null;
      let paused = false;
      const DEFAULT_DELAY = 100;  // Default delay in ms for non-ride features.
      let startEndMarkers = [];
      let rideNumbers = [];
      let descriptions = [];
      let animationDelays = []; // Array to hold per-coordinate delay in ms.

      const daySelect = document.getElementById("daySelect");
      const rideNumberDisplay = document.getElementById("rideNumberDisplay");
      const descriptionDisplay = document.getElementById("descriptionDisplay");
      const startButton = document.getElementById("startButton");
      const pauseButton = document.getElementById("pauseButton");
      const resumeButton = document.getElementById("resumeButton");

      // Load available days into the dropdown.
      async function loadAvailableDays() {
        daySelect.innerHTML = days.map(day => `<option value="${day}">${day}</option>`).join("");
      }

      // Returns HTML description for a feature.
      function getDescription(feature) {
        const tipValue = parseFloat(feature.properties.tip);
        // If tip is non-zero, set background to light green; otherwise, no extra background.
        const bgStyleTip = tipValue > 0 ? 'background-color: lightgreen;' : '';
        const surgeValue = parseFloat(feature.properties.surge);
        const bgStyleSurge = surgeValue > 0 ? 'background-color: #FED8B1;' : '';
        return `
          <div class="popup-content">
            <h3>${feature.properties.rideType}</h3>
            <p><strong><span class="popup-icon">📍</span>Pickup:</strong> ${feature.properties.pickupAddress}</p>
            <p><strong><span class="popup-icon">🚩</span>Dropoff:</strong> ${feature.properties.dropoffAddress}</p>
            <hr>
            <p><strong><span class="popup-icon">💵</span>Base Pay:</strong> $${feature.properties.basePay}</p>
            <p style="${bgStyleSurge}"><strong><span class="popup-icon">⚡</span>Surge:</strong> $${feature.properties.surge}</p>
            <p style="${bgStyleTip}"><strong><span class="popup-icon">🤝</span>Tip:</strong> $${feature.properties.tip}</p>
            <p><strong><span class="popup-icon">💰</span>Earnings:</strong> $${feature.properties.earnings}</p>
            <hr>
            <p><strong><span class="popup-icon">📏</span>Distance:</strong> ${feature.properties.distance} miles</p>
            <p><strong><span class="popup-icon">⏱</span>Duration:</strong> ${feature.properties.duration} sec</p>
            <hr>
            <p><strong><span class="popup-icon">⏰</span>Start:</strong> ${feature.properties.rideStartTime}</p>
            <p><strong><span class="popup-icon">🏁</span>End:</strong> ${feature.properties.rideEndTime}</p>
          </div>
        `;
      }

      // Load GeoJSON and extract coordinates, ride numbers, descriptions, and compute delays.
      async function loadGeoJSON(day) {
        // Append a cache-busting query parameter.
        const url = `output/geojson/${day}.geojson?t=${new Date().getTime()}`;
        const response = await fetch(url);
        const geojson = await response.json();
        const coords = [];
        rideNumbers = [];
        descriptions = [];
        animationDelays = [];
        geojson.features.forEach((feature, i) => {
          if (feature.geometry.type === "LineString") {
            const rideNum = `${i + 1}`;
            let desc;
            // Compute a per-coordinate delay if feature is a ride.
            let delayValue = DEFAULT_DELAY;
            if (feature.properties.type === "ride" && feature.properties.duration) {
              const durationSeconds = parseFloat(feature.properties.duration); // duration in seconds
              // Total animation time for this feature (in ms) = (duration/400)*1000.
              const totalTimeMs = (durationSeconds / 400) * 1000;
              const numCoords = feature.geometry.coordinates.length;
              delayValue = totalTimeMs / numCoords;
            }
            // Otherwise, use a default delay for non-ride features.
            desc =
              feature.properties.type === "ride"
                ? getDescription(feature)
                : "<strong>Racing to my next ride! 🏎️</strong>";

            feature.geometry.coordinates.forEach((c) => {
              coords.push(c);
              rideNumbers.push(rideNum);
              descriptions.push(desc);
              animationDelays.push(delayValue);
            });
          }
        });
        return { geojson, coords };
      }

      // Add start/end markers with popups for ride features.
      function addStartEndMarkers(geojson) {
        // Remove existing markers.
        startEndMarkers.forEach((m) => m.remove());
        startEndMarkers = [];

        geojson.features.forEach((feature) => {
          if (feature.geometry.type === "LineString" && feature.properties.type === "ride") {
            const coords = feature.geometry.coordinates;
            if (coords.length < 2) return;

            // Create start marker (green dot).
            const startEl = document.createElement("div");
            startEl.textContent = "🟢";
            startEl.style.fontSize = "16px";
            startEl.addEventListener("mouseenter", () => {
              const popup = new mapboxgl.Popup({
                closeButton: false,
                closeOnClick: false
              })
                .setLngLat(coords[0])
                .setHTML(getDescription(feature))
                .addTo(map);
              startEl._popup = popup;
            });
            startEl.addEventListener("mouseleave", () => {
              if (startEl._popup) {
                startEl._popup.remove();
                startEl._popup = null;
              }
            });
            const startMarker = new mapboxgl.Marker({ element: startEl })
              .setLngLat(coords[0])
              .addTo(map);

            // Create end marker (red dot).
            const endEl = document.createElement("div");
            endEl.textContent = "🛑";
            endEl.style.fontSize = "16px";
            endEl.addEventListener("mouseenter", () => {
              const popup = new mapboxgl.Popup({
                closeButton: false,
                closeOnClick: false
              })
                .setLngLat(coords[coords.length - 1])
                .setHTML(getDescription(feature))
                .addTo(map);
              endEl._popup = popup;
            });
            endEl.addEventListener("mouseleave", () => {
              if (endEl._popup) {
                endEl._popup.remove();
                endEl._popup = null;
              }
            });
            const endMarker = new mapboxgl.Marker({ element: endEl })
              .setLngLat(coords[coords.length - 1])
              .addTo(map);

            startEndMarkers.push(startMarker, endMarker);
          }
        });
      }

      // Start animation: load data, add markers, and begin the animation loop.
      function startAnimation() {
        cancelAnimationFrame(animationFrameId);
        if (marker) { marker.remove(); }
        animationIndex = 0;
        paused = false;
        const selectedDay = document.getElementById("daySelect").value;

        fetch(`output/geojson/${selectedDay}.geojson`)
          .then(response => response.json())
          .then((geojson) => {
            // Add start/end markers.
            addStartEndMarkers(geojson);
            return loadGeoJSON(selectedDay);
          })
          .then(({ coords }) => {
            animationCoords = coords;
            if (!animationCoords.length) return;

            // Create an animated marker (a car).
            const carEl = document.createElement("div");
            carEl.textContent = "🚙";
            carEl.style.fontSize = "24px";
            carEl.style.transition = "transform 0.1s linear";
            carEl.style.willChange = "transform";
            marker = new mapboxgl.Marker({ element: carEl })
              .setLngLat(animationCoords[0])
              .addTo(map);

            const routeGeoJSON = {
              type: "Feature",
              geometry: {
                type: "LineString",
                coordinates: animationCoords,
              },
            };

            map.getSource("route").setData(routeGeoJSON);
            map.fitBounds(turf.bbox(routeGeoJSON), { padding: 20 });
            animate();
          });
      }

      // Animation loop uses per-coordinate delay.
      function animate() {
        if (paused) return;
        if (animationIndex < animationCoords.length) {
          marker.setLngLat(animationCoords[animationIndex]);
          document.getElementById("rideNumberDisplay").textContent =
            rideNumbers[animationIndex];
          document.getElementById("descriptionDisplay").innerHTML =
            descriptions[animationIndex];
          // Get delay for the current coordinate or use default.
          const delay = animationDelays[animationIndex] || DEFAULT_DELAY;
          animationIndex++;
          animationFrameId = setTimeout(() => requestAnimationFrame(animate), delay);
        }
      }

      function pauseAnimation() {
        paused = true;
        cancelAnimationFrame(animationFrameId);
      }

      function resumeAnimation() {
        if (!paused) return;
        paused = false;
        requestAnimationFrame(animate);
      }

      // On map load, set up sources and layers; then load available days and start animation.
      map.on("load", async () => {
        map.addSource("route", {
          type: "geojson",
          data: {
            type: "Feature",
            geometry: {
              type: "LineString",
              coordinates: [],
            },
          },
        });

        map.addLayer({
          id: "route-line",
          type: "line",
          source: "route",
          paint: {
            "line-color": "#007cbf",       // A pleasant blue.
            "line-width": 3,               // A bit thinner than 4px.
            "line-dasharray": [1, 3],       // Creates a dashed effect.
            "line-opacity": 0.8            // Slight transparency.
          }
        });

        await loadAvailableDays();
        // Choose a random day.
        const randomIndex = Math.floor(Math.random() * days.length);
        const randomDay = days[randomIndex];
        document.getElementById("daySelect").value = randomDay;
        startAnimation();
        document.getElementById("daySelect").addEventListener("change", startAnimation);
      });

      // Bind buttons.
      document.getElementById("startButton").addEventListener("click", startAnimation);
      document.getElementById("pauseButton").addEventListener("click", pauseAnimation);
      document.getElementById("resumeButton").addEventListener("click", resumeAnimation);
    </script>
  </body>
</html>

Conclusion

Vibe coding to make your idea come to life is definitely possible, but it does take some programming language fluency to go beyond the initial version and get exactly what you want. While the end result will be dazzling, I think the biggest benefit is that you get to learn by example. Now I know a lot more about GeoJSON, the Mapbox API, web animation, caching API requests, and CSS styling. I'll be able to take these skills and apply them to future projects, thinking back to how they applied to this use case.

As with any programming process, you will inevitably hit road blocks and frustrations. One of the biggest benefits of AI-assistance is to get past these frustrations. Whereas this would previously only involve language/library references, frenzied google searching, and combing through StackOverflow posts, now we can get more specific help on our exact coding quandaries.

There is clearly code smell to this AI-assisted code though. There's inconsistent formatting, commenting, and the structure is somewhat bizarre. Any seasoned coder would be able to tell this is vibe-coded. Don't make the mistake of citing the code as entirely your own. While this isn’t stack-overflow-perfect code, it’s yours in spirit. AI helped build the skeleton, but you made it move.

I hope this was a revealing expose on how to leverage AI to make that thing you want. If you’re looking for an innovative thinker to join your team, or you’re a fellow creative who’s interested in vibe coding one of your ideas, I’d love to hear from you!

Full source code and data files for this project can be found here: github repo

📩 Drop me a note here:
LinkedIn | Github | Email

How I Hacked Uber’s Hidden API to Download 4379 Rides

Nicholas Bennett — Thu, 10 Apr 2025 01:07:47 +0000

When companies build walls around your own data, you have two choices: wait and hope, or take control.

I chose control.

In this post, I’ll show you how I reverse-engineered Uber’s internal APIs to scrape my driver earnings data programmatically with zero API key, how I cleaned it up to be useful for downstream data analysis, and then how I was able to derive actionable insights from that data.

Between January 9 2023 and September 30 2024 I drove with Uber sometimes 7 days a week for a total of 4379 passenger rides. I had over 2500 rides with 5 star ratings and I averaged 4.98 stars. Being an analytical person I wanted to learn from my experience. With such a data-driven app, I thought surely there’d be a way to get my data about my driving.

However, when I looked at the Drivers API on their site, I got the rude message that the API was “limited” and I had to apply for access. Seriously?! There must be another way.

I found another page (https://drivers.uber.com/earnings/activities) that would let me look at my past rides, but frustratingly in a paginated week-by-week view. I had driven Uber for roughly 89 weeks, and with 15 results per page I’d be clicking at least 400 times just to list my rides. I thought there had to be a way around this.

Hacking The Data

I opened the Google Chrome Developer Tools, went to the Network tab, and refreshed the page on a week that had ride data. I was looking for a request that was providing the data for the listing of rides.

I found it! The request was being made to https://drivers.uber.com/earnings/api/getWebActivityFeed?localeCode=en with some request parameters in a POST header, and I was getting back a JSON response with details far richer than what was being displayed in the browser. The pagination flags were easy to find at the end of the JSON response. My task then was to replicate this request programmatically in Python, my language of choice for web scraping.

To make things easier for myself, I leaned on Postman’s code generation capabilities. I first had to copy this Uber API request over from Google Chrome to Postman and replicate it there. There was surely going to be some authentication built in to the request, and instead of monkeying around with manually re-creating that I just used Google Chrome Dev Tool’s “Copy > Copy as cURL” on the request in the Network tab:

This copies to the clipboard a full cURL command with all parameters needed to replicate this on the command line. Postman can import a cURL command and generate the request in its UI:

Wow! I was able to transfer the request to Postman, got it to work there, and it generated some Python code using the requests library that I could easily copy over to my code editor and continue to work on there.

Next step was to add support for pagination and ranging between the start date (1/9/2023) and end date (9/30/2024) so I could get all of my ride data. I created a new folder for my project, set up a Python3 virtualenv, installed Jupyter notebook and requests, and created a Jupyter notebook with the code to range through the pages and dates (full code here). The main chunk of the scraping code is below, with the security tokens redacted for my specific request:

from datetime import datetime, timedelta
import json
import requests
class UberDriver:
    def __init__(self):
        self.headers = {
          'accept': '*/*',
          'accept-language': 'en-US,en;q=0.9',
          'content-type': 'application/json',
          'cookie': 'REDACTED',
          'origin': 'https://drivers.uber.com',
          'priority': 'u=1, i',
          'referer': 'https://drivers.uber.com/earnings/activities',
          'sec-ch-ua': '"Not A(Brand";v="8", "Chromium";v="132", "Google Chrome";v="132"',
          'sec-ch-ua-mobile': '?0',
          'sec-ch-ua-platform': '"macOS"',
          'sec-fetch-dest': 'empty',
          'sec-fetch-mode': 'cors',
          'sec-fetch-site': 'same-origin',
          'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36',
          'x-csrf-token': 'x',
          'x-uber-earnings-seed': 'REDACTED'
        }

    def get_rides(self, start_date_iso, end_date_iso):        
        url = "https://drivers.uber.com/earnings/api/getWebActivityFeed?localeCode=en"

        payload = json.dumps({
          "startDateIso": start_date_iso,
          "endDateIso": end_date_iso,
          "paginationOption": {}
        })

        response = requests.request("POST", url, headers=self.headers, data=payload)
        data = response.json()

        rides = data['data']['activities']
        while data['data']['pagination']['hasMoreData']:
            payload = json.dumps({
              "startDateIso": start_date_iso,
              "endDateIso": end_date_iso,
              "paginationOption": {"cursor": data['data']['pagination']['nextCursor']}
            })
            response = requests.request("POST", url, headers=self.headers, data=payload)
            data = response.json()
            if data['data']['activities']:
                rides = rides + data['data']['activities']
        return rides or []

    def get_ride_detail(self, ride_uuid):
        # This is only helpful to get additional fare breakdown from Uber, if we wanted to analyze how much
        # Uber is taking from each fare.
        url = f"https://drivers.uber.com/earnings/trips/{ride_uuid}"
        response = requests.request("GET", url, headers=self.headers)
        return response.text

uber = UberDriver()

# This is the date I started working as an Uber driver; modify for your start date
start_date = datetime.strptime("2023-01-09", "%Y-%m-%d")
end_date = datetime.today()
current_date = start_date

rides = []
while current_date <= end_date:
    next_date = current_date + timedelta(days = 7)
    start_date_iso = current_date.strftime('%Y-%m-%d')
    end_date_iso = next_date.strftime('%Y-%m-%d')
    print(f"Getting rides for {start_date_iso} - {end_date_iso}...")
    new_rides = uber.get_rides(start_date_iso, end_date_iso)
    print(f"Retrieved {len(new_rides)} rides.")
    rides += new_rides
    current_date = next_date

# Let's dump all of these rides to a JSON file so we can reference this data outside of the script if need be, 
# or simply not have to retrieve from Uber again.
with open(f"data/rides_raw.json", "w") as file:
    json.dump(rides, file)

print(f"Retrieved {len(rides)} rides total.")

At the end, I was able to get all of my rides data! It took about 10 minutes to download it all, and this is entirely without delays in my code (e.g. import time; time.sleep(0.1)). Typically on a web scraping project I would add in some waits between requests to avoid hammering the API and hitting throttling/rate limiting, but in this case I was able to get away with it.

The total file size for my 4379 rides was 4.9 MB (rides_raw.json).

Cleaning The Data

Upon further inspection, I found the data contained a lot of cruft clearly meant for just rendering the data on the web page. It was not meant for public consumption.

{
  "uuid": "d3096d6c-02bd-4f8e-855b-117588b27910",
  "recognizedAt": 1673809557,
  "activityTitle": "Comfort",
  "formattedTotal": "$10.72",
  "routing": {
    "webviewUrl": "https://drivers.uber.com/earnings/trips/d3096d6c-02bd-4f8e-855b-117588b27910",
    "deeplinkUrl": null
  },
  "breakdownDetails": { "formattedTip": "$1.00", "formattedSurge": null },
  "tripMetaData": {
    "formattedDuration": "15 min 56 sec",
    "formattedDistance": "3.9 mi",
    "pickupAddress": "N Ashland Ave, Chicago, IL 60614-1105, US",
    "dropOffAddress": "W Madison St, Chicago, IL 60612, US",
    "mapUrl": "https://static-maps.uber.com/map?width=360&height=100&marker=lat%3A41.92946%24lng%3A-87.66817%24icon%3Ahttps%3A%2F%2Fd1a3f4spazzrp4.cloudfront.net%2Fmaps%2Fhelix%2Fcar-pickup-pin.png%24anchorX%3A0.5%24anchorY%3A0.0&marker=lat%3A41.88207%24lng%3A-87.67135%24icon%3Ahttps%3A%2F%2Fd1a3f4spazzrp4.cloudfront.net%2Fmaps%2Fhelix%2Fcar-dropoff-pin.png%24anchorX%3A0.5%24anchorY%3A0.0&polyline=color%3A0xFF2DBAE4%24width%3A4%24enc%3Aaj%7C%7EF%60uavOyNV%7Ep%40O%60N%60Bp%5BuDn%7CEsD%5C%60Tc%40L%3FhE_B%40&signature=jOjTSurxb6QZGZQlr-tHwSUpbYc="
  },
  "type": "TRIP",
  "status": "COMPLETED"
}

A few problems:

The fare, surge, tip, distance, and duration for the ride was in a human-readable format, which is not helpful for numeric comparisons.
The data was irregular, sometimes containing a field and sometimes having a null value. For example, if there was no tip or surge value, then breakdownDetails was null.
I didn’t need the webviewUrl, deeplinkUrl, or mapUrl, and the uuid was only useful internally to Uber.
The recognizedAt seemed to be a UNIX timestamp indicating when the ride started, and I would have to extrapolate from that and the formattedDuration when the ride ended.
The pickupAddress and dropOffAddress were very general (clearly for safety purposes), and the ZIP was probably the most useful part.

As with any web scraping or data processing project, I had to write a fair amount of code to clean this up and shape it into a format I needed for further analysis. I used a combination of Pandas and regular expressions to clean it up (full code here).

from datetime import datetime, timedelta
import re
import pandas as pd

rides_df = pd.read_json('../data/rides_raw.json')

# We need that sweet sweet trip metadata so drop any records that don't have it
rides_df = rides_df[rides_df['tripMetaData'].notnull()]
# Drop the useless routing and uuid columns
rides_df.drop(['routing', 'uuid'], axis=1, inplace=True)

# Count how many records have each number of keys and sort by key count
num_breakdown_keys = rides_df['breakdownDetails'].apply(lambda x: len(x) if isinstance(x, dict) else 0)
print(num_breakdown_keys.value_counts().sort_index())

rides_with_breakdown = rides_df[rides_df['breakdownDetails'].notnull()]
# I think this is only included if either the tip or surge value is non-zero. 
# Let's see if any records have zero for both.
breakdown_zero_valued = rides_with_breakdown['breakdownDetails'].apply(\
    lambda d: (d['formattedTip'] is None or d['formattedTip'] == '$0.00') \
            and (d['formattedSurge'] is None or d['formattedSurge'] == '$0.00'))
print(f"Number of records where both formattedTip and formattedSurge are zero or None: {breakdown_zero_valued.sum()}")

def fix_breakdown(bd):
    # If breakdownDetails is None or missing, return a default dictionary.
    if bd is None or pd.isnull(bd):
        return {'formattedTip': '$0.00', 'formattedSurge': '$0.00'}
    # Otherwise, if it's a dictionary, fix the individual keys.
    if isinstance(bd, dict):
        if bd.get('formattedTip') is None:
            bd['formattedTip'] = '$0.00'
        if bd.get('formattedSurge') is None:
            bd['formattedSurge'] = '$0.00'
    return bd

rides_df['breakdownDetails'] = rides_df['breakdownDetails'].apply(fix_breakdown)

def parse_currency_to_float(currency_str):
    clean_str = currency_str.replace('$', '').strip()
    return float(clean_str)

rides_df['tip'] = rides_df['breakdownDetails'].apply(lambda bd: parse_currency_to_float(bd['formattedTip']))
rides_df['surge'] = rides_df['breakdownDetails'].apply(lambda bd: parse_currency_to_float(bd['formattedSurge']))
rides_df.drop('breakdownDetails', axis=1, inplace=True)

def parse_time_to_seconds(time_str):
    matches = re.findall(r'(\d+)\s*(hr|min|sec)', time_str)
    unit_to_seconds = {'hr': 3600, 'min': 60, 'sec': 1}
    return sum(int(value) * unit_to_seconds[unit] for value, unit in matches)

def parse_miles(miles_str):
    match = re.search(r'(\d+\.?\d*)\s*mi', miles_str)
    if match:
        return float(match.group(1))

def extract_zipcode(address):
    zip_code_pattern = re.compile(r'\b\d{5}\b')
    match = zip_code_pattern.search(address)
    if match:
        return match.group()

rides_df['duration'] = rides_df['tripMetaData'].apply(lambda md: parse_time_to_seconds(md['formattedDuration']))
rides_df['distance'] = rides_df['tripMetaData'].apply(lambda md: parse_miles(md['formattedDistance']))
rides_df['pickup_address'] = rides_df['tripMetaData'].apply(lambda md: md['pickupAddress'])
rides_df['dropoff_address'] = rides_df['tripMetaData'].apply(lambda md: md['dropOffAddress'])
rides_df['pickup_zip'] = rides_df['pickup_address'].apply(lambda a: extract_zipcode(a))
rides_df['dropoff_zip'] = rides_df['dropoff_address'].apply(lambda a: extract_zipcode(a))
rides_df.drop('tripMetaData', axis=1, inplace=True)

rides_df['earnings'] = rides_df['formattedTotal'].apply(lambda t: parse_currency_to_float(t))
rides_df['base_pay'] = rides_df['earnings'] - rides_df['surge'] - rides_df['tip']
rides_df.drop('formattedTotal', axis=1, inplace=True)

rides_df['ride_start'] = rides_df['recognizedAt'].apply(lambda t: datetime.fromtimestamp(t))
rides_df['ride_end'] = rides_df['ride_start'] + rides_df['duration'].apply(lambda s: timedelta(seconds=s))
rides_df.drop('recognizedAt', axis=1, inplace=True)

rides_df = rides_df.rename(columns={'activityTitle': 'ride_type', 'type': 'ride_category'})

# I know these are the ride types (activityTitle) I'm interested in
passenger_ride_types = ['Business Comfort', 'Comfort', 'Share', 'Uber Pet',\
                        'UberX', 'UberX Priority', 'UberX Share', 'UberXL']
ride_categories = ['CT', 'TRIP']
print(f"Total passenger rides: {len(rides_df[rides_df['ride_type'].isin(passenger_ride_types)])}")
# How many completed passenger rides do we have?
rides_df = rides_df.query('status=="COMPLETED" and ride_category in @ride_categories and ride_type in @passenger_ride_types')
print(f"There are {len(rides_df)} completed passenger rides in this data.")

rides_df['id'] = rides_df.index
rides_df.to_csv('../data/rides.csv', index=False)

The crown jewel of all of this was a 796 KB CSV file with the data in consistent, easily digestible formats (rides.csv). Here are a few of the cleaned records:

ride_type,ride_category,status,tip,surge,duration,distance,pickup_address,dropoff_address,pickup_zip,dropoff_zip,earnings,base_pay,ride_start,ride_end,id
Comfort,TRIP,COMPLETED,1.0,0.0,956,3.9,"N Ashland Ave, Chicago, IL 60614-1105, US","W Madison St, Chicago, IL 60612, US",60614,60612,10.72,9.72,2023-01-15 14:05:57,2023-01-15 14:21:53,0
UberX,TRIP,COMPLETED,0.0,0.0,835,2.4,"N Lincoln Ave, Chicago, IL 60613, US","N Southport Ave, Chicago, IL 60614, US",60613,60614,4.13,4.13,2023-01-15 13:45:31,2023-01-15 13:59:26,1
UberX,TRIP,COMPLETED,3.0,0.0,642,1.6,"N Clifton Ave, Chicago, IL 60657-2224, US","N Lincoln Ave, Chicago, IL 60613, US",60657,60613,7.56,4.56,2023-01-15 13:33:41,2023-01-15 13:44:23,2
UberX,TRIP,COMPLETED,3.0,3.75,713,1.5,"W Belmont Ave, Chicago, IL 60657-4511, US","3001 N Ashland Ave, Chicago, IL 60657, US",60657,60657,11.19,4.4399999999999995,2023-01-15 13:17:41,2023-01-15 13:29:34,3

Insights

I've been using this in many ways to look back on the time I spent driving for Uber around Chicago at all hours of the day, in all different neighborhoods, in all types of weather, on all days of the week, and picking up all ages of people (and some cats, dogs, and bunnies too 🐇).

To start, let me show you some things I learned about strategically making more

💰MONEY💰

With Uber driving, outside of surge pricing, it's all about the tips. I wanted to get insights on what factors lead to maximal tips.

What time of day should I drive to maximize tips?
What day of the week should I drive to maximize tips?
What ride type (UberX, Comfort, Pet, etc) lead to maximal tips?
What are the top 5 ZIP codes to pick up from to maximize tips?

For each day I totaled the tips on the hour of day (e.g. 4:00 - 4:59 PM), then found the average of total tips across all days and divided by the number of days to find the average total tips per hour of day.

Below are the key insights. If you want to see the Python code I used to do this analysis and generate the charts using Seaborn (which makes charting easier for me than plain Matplotlib), you can find my full analysis Jupyter notebook on my Github repo here: Tip Analysis.ipynb

Caveats: This is based on the times and days I drove, and I didn't drive every hour of every day. This also doesn't take into account outliers, like if someone gave me a $50 tip one night.

1. What time of day should I drive to maximize tips?

The best time of day overall was 1:00 - 1:59 AM with an average total tip per hour of $4.383514.

2. What day of the week should I drive to maximize tips?

The best day to drive was Saturday, with an average total tip per hour of $3.758764.

3. What ride type (UberX, Comfort, Pet, etc) lead to maximal tips?

The best ride type was UberX Priority, with an average total tip per hour of $4.314286. Since this ride type is not selectable in the Uber Driver app (it comes up occasionally as an offering when driving UberX), based on the chart you can see that the next best ride type is Comfort.

4. What are the top 5 ZIP codes to pick up from to maximize tips?

The top 5 ZIP codes to pick up from to maximize average total tip per hour were:

    pickup_zip  avg_tip_total
       60453      13.980000
       60160       9.526667
       60069       8.790000
       60461       8.230000
       60126       7.075000

Beyond Tips

To give you a little more insight into the data beyond the tips, here are some more charts again using Seaborn.

P.S. - The code I used to generate that heatmap up there was with Folium using this script: Zip Code Heatmap

You can interact with it live here: chicago_map.html

Conclusion

In the end, I didn’t let Uber’s limited interface hold me back. I took control. By reverse-engineering hidden endpoints and automating the extraction of thousands of rides, I transformed a mess of paginated data into an easy-to-use 796 KB CSV that tells the real story of my driving history. This isn't just about web scraping. It’s about demonstrating how to relentlessly pursue actionable insights even when the system won’t hand you the answers.

Using simple tools like Google Chrome Developer Tools, Postman, and Python, I turned an overwhelming set of fragmented records into a clean, coherent dataset that properly reflects my driving experience. The journey from tedious clicks to a robust dataset is a testament to my ability to innovate and overcome data barriers, skills that I bring to every challenge I face.

If you’re a hiring manager looking for someone who thinks creatively about ways to overcome complex obstacles, or a fellow driver seeking new ways to understand your performance, I’d love to talk. Let’s transform data frustrations into opportunities together. Thanks for reading, and I look forward to continuing the conversation.

If you appreciate a hands-on, data-driven approach and believe in seizing control, let’s connect. I’m always open to exploring new challenges and opportunities.

Contact Me:

LinkedIn | Email | GitHub

DEV Community: Nicholas Bennett

Promptagogy: Beyond Prompt Engineering

A New Approach to AI Literacy

The Complement to Current Approaches

Promptagogy in Action: Real-World Examples

From Technical Complexity to Vivid Understanding

My Journey to Promptagogy

Early Learning Experiences

Professional Evolution

Finding Models for Adaptive Learning

Epistemological Pluralism in the Age of AI

The Five Principles of Promptagogy

1. Observation Alongside Instruction

2. Process As Valuable As Product

3. Diversity of Cognitive Approaches

4. Crystallized Thinking

5. Collective Learning

Finding Balance: When to Use Each Approach

Promptagogy in Practice: A Tool for Collective Learning

The Process Behind This Article

The Meta-Learning Revolution

Promptagogy for Different Fields

For Educators

For Developers

For Researchers

Join the Promptagogy Movement

Further Reading

Web Scraping Unwrapped: From Python Beginner to Data Harvester

Introduction

Table of Contents

My Background

How I Scrape

Python: requests

Python: requests + json/csv/pandas

Python: requests + Beautiful Soup

Python with a Headless Browser: Selenium

Anti-Scraping Technology

IP Address

User Agent Filtering / HTTP Headers

CAPTCHAs / CloudFlare

Honeypots

Behavior Analysis

The Ethics of Web Scraping

Terms of Service: The Digital Fine Print

Rate Limiting: Don't Be a Resource Hog

The Data Privacy Angle

Robots.txt: The Digital "No Trespassing" Sign

The "Be Reasonable" Test

Giving Back to the Community

Resources

Conclusion

Vibe Coding 4,379 Uber Rides Into a Living Map

Introduction

The Vibe Coding Process

The Challenge

Solution Exploration with ChatGPT

Step 1: Geocoding and Routing with Mapbox

Step 2: Building Interactive Map with Mapbox GL JS and Turf.js

Conclusion

How I Hacked Uber’s Hidden API to Download 4379 Rides

Hacking The Data

Cleaning The Data

Insights

💰MONEY💰

1. What time of day should I drive to maximize tips?

2. What day of the week should I drive to maximize tips?

3. What ride type (UberX, Comfort, Pet, etc) lead to maximal tips?

4. What are the top 5 ZIP codes to pick up from to maximize tips?

Beyond Tips

Conclusion