I Was Spending 48 Hours Every Month Checking Supplier Prices. So I Built a Python Script to Do It for Me

About a year ago I realized I had accidentally turned myself into a very expensive copy and paste machine.

Every morning started the same way. Open supplier website. Search for the product SKU. Find the correct variation. Copy the price into a spreadsheet. Move to the next supplier and repeat.

At first it didn't seem like a big deal.

Then I counted how much time I was actually spending on it.

I was tracking around 25 products across 15 suppliers. On average it took about 90 seconds to find and verify a single price. Sometimes less, sometimes more if the website was slow or the product had multiple variations.

When I added everything up, the result was painful.

I was spending almost 48 hours every month checking prices manually.

That is more than an entire work week spent moving numbers from websites into Excel.

At some point I had to ask myself a simple question:

Why am I doing work that a script could probably do better than me?

So I opened my editor and started writing Python.

As you can probably guess, it did not work on the first attempt.

Why this mattered

Margins in ecommerce can disappear surprisingly fast.

If one supplier increases prices by 7 percent and you don't notice for two weeks, you're making decisions based on outdated information.

You order inventory at the wrong price.

You run promotions with the wrong assumptions.

You think your margins are healthy when they actually aren't.

The real problem wasn't the time I spent checking prices.

The real problem was that I never fully trusted the data I was working with.

Some suppliers updated prices every few days.

Others updated them once a month.

A few changed prices whenever raw material costs changed.

I always felt like I was looking in the rear view mirror instead of seeing what was happening right now.

The first version

Like most first versions, mine was built on optimism.

import requests

response = requests.get(url)
print(response.text)

In my head, the script would visit the page, grab the price and save it somewhere.

Problem solved.

Reality had other ideas.

Some suppliers loaded prices with JavaScript.

Some blocked scraping requests.

One supplier changed their HTML structure so often that I started recognizing their frontend updates before their customers did.

Another required login and two factor authentication before showing prices.

Because of course they did.

After a few rounds of trial and error I ended up with a combination of tools:

• requests for simple pages
• BeautifulSoup for parsing HTML
• Selenium for websites that relied heavily on JavaScript
• pandas for storing and comparing price history

Was it elegant?

Not really.

Did it work?

Most of the time, yes.

And honestly, production code and portfolio code are usually very different things.

The final approach

The final script followed a pretty simple flow.

First, load the list of products and supplier URLs.

Then visit each page and extract the current price.

Finally, store the results and compare them against previous data.

The core looked something like this:

import pandas as pd
import requests
from bs4 import BeautifulSoup
from datetime import datetime

results = []

for sku, product in PRODUCTS.items():
    try:
        response = requests.get(
            product["url"],
            timeout=10,
            headers={
                "User-Agent": "Mozilla/5.0"
            }
        )

        soup = BeautifulSoup(
            response.text,
            "html.parser"
        )

        price_element = soup.select_one(
            ".product-price"
        )

        price = float(
            price_element.text
            .replace("$", "")
            .replace(",", "")
        )

        results.append({
            "sku": sku,
            "supplier": product["supplier"],
            "price": price,
            "timestamp": datetime.now()
        })

    except Exception as e:
        print(f"Failed for {sku}: {e}")

The interesting part wasn't collecting prices.

Collecting data is easy.

Knowing when something important changes is where the value comes from.

Detecting changes

Every day the script compared the latest prices with yesterday's data.

old_prices = pd.read_csv(
    "yesterday_prices.csv"
)

new_prices = pd.read_csv(
    "today_prices.csv"
)

changes = new_prices.merge(
    old_prices,
    on="sku",
    suffixes=(
        "_new",
        "_old"
    )
)

changes["difference"] = (
    changes["price_new"] -
    changes["price_old"]
)

alerts = changes[
    abs(changes["difference"]) > 0.5
]

print(alerts)

Instead of manually reviewing hundreds of prices, I only had to look at the handful that had actually changed.

That small shift completely changed the workflow.

Instead of searching for information, I was reacting to information that mattered.

The problems nobody talks about

Building automation doesn't remove work.

It changes the type of work you do.

For example, suppliers love redesigning websites.

One day the price lives inside:

<div class="price">

The next day it suddenly becomes:

<span class="cost">

Your parser breaks.

Your script happily collects nothing.

You discover the issue three days later.

Good times.

Another problem was temporary outages.

Sometimes a supplier website would fail to load and the script interpreted the result as:

Price = 0

Receiving an alert saying that every product in your catalog suddenly became free is exciting for about three seconds.

Then reality catches up.

There was also anti bot protection.

One supplier started rate limiting requests, so I ended up adding randomized delays between requests.

import random
import time

time.sleep(
    random.uniform(2, 5)
)

Not sophisticated.

But effective enough.

The results

After running the system for a few months, the numbers were pretty clear.

Manual price checks dropped from around 48 hours per month to roughly 7.

Most of those seven hours were spent validating unusual results instead of searching for prices.

The script detected:

• 126 supplier price changes
• 19 opportunities to switch suppliers
• 7 products where margins were shrinking without anyone noticing

The estimated savings were around:

• 41 hours every month
• $2,800 per year in purchasing costs
• An unmeasurable amount of frustration

The time savings were nice.

The visibility was even more valuable.

For the first time I felt confident that business decisions were based on current information instead of outdated spreadsheets.

What I would do differently

If I were rebuilding this today, I would change three things.

First, I would use APIs whenever possible.

Websites change.

APIs usually don't.

If a supplier gives you an API, take it and don't look back.

Second, I would add monitoring from day one.

The first version failed silently more than once.

Silent failures are dangerous because everything looks fine until you realize you've been collecting bad data for a week.

Finally, I would spend less time trying to make the code beautiful.

The original version was messy.

There were duplicated functions, questionable decisions and more than a few things I wouldn't show during a code review.

But it worked.

Sometimes ugly code running in production creates more value than perfect architecture sitting in a private Git repository.

The bigger lesson

This story isn't really about scraping websites.

It's about recognizing repetitive work.

Every business has tasks that slowly become invisible because you've done them so many times.

Copying numbers.

Renaming files.

Moving data between systems.

Generating reports.

Eventually those tasks stop feeling inefficient and start feeling normal.

That's usually the moment when automation becomes interesting.

You don't need to be a software engineer to benefit from it.

You just need to notice when you're solving the same problem for the fiftieth time.

The code is often the easy part.

Recognizing what deserves automation is usually much harder.

TL;DR

• I manually checked prices across 15 suppliers.
• The process consumed around 48 hours every month.
• I built a Python script using requests, BeautifulSoup, pandas and Selenium.
• The script monitored prices automatically and only reported changes.
• Manual work dropped to around 7 hours per month.
• Estimated savings were roughly $2,800 per year.
• The biggest benefit wasn't saving time.
• It was finally being able to make decisions using current data.

And yes, I probably spent more time building the automation than I saved during the first month.

Developers have a habit of doing that.

Fortunately, this turned out to be one of those projects that actually paid for itself.

What repetitive task in your work have you accepted as normal even though a script could probably do it for you?