Alex Spinov

Posted on Mar 25

The Side Project That Taught Me More Than 4 Years of College

#programming #discuss #beginners #career

College taught me algorithms. My side project taught me everything else.

In 2023, I decided to build a web scraping platform. Not because anyone asked me to — I just wanted to scrape Reddit data for a personal project.

3 years later, I have:

77 production scrapers running daily
500+ technical articles published
290+ open source repositories
A consultancy that builds data pipelines for companies

Here's what I learned that no CS class ever covered.

1. Production code is 20% logic, 80% error handling

In college, we wrote functions that received perfect input and returned correct output. In production:

The API returns 429 (rate limited) at 3 AM
The HTML structure changes without warning
Memory spikes to 4GB because one page had 10,000 comments
The proxy gets banned in Brazil but works in Germany

# College version
def scrape(url):
    return requests.get(url).json()

# Production version
def scrape(url, retries=3, timeout=10):
    for attempt in range(retries):
        try:
            resp = requests.get(url, timeout=timeout, headers=HEADERS)
            resp.raise_for_status()
            return resp.json()
        except requests.exceptions.HTTPError as e:
            if resp.status_code == 429:
                time.sleep(2 ** attempt)
                continue
            raise
        except requests.exceptions.ConnectionError:
            if attempt == retries - 1:
                raise
            time.sleep(1)
    return None

2. Nobody cares about your tech stack

I spent weeks choosing between FastAPI and Flask, PostgreSQL and MongoDB, Docker and bare metal.

My users don't know or care what's running behind the scenes. They care that:

It returns data in <5 seconds
It doesn't crash
The output format makes sense

I could have shipped 2 weeks earlier if I'd just picked the boring option.

3. Documentation IS the product

The difference between a scraper with 0 users and one with users? A good README.

What does it do? (one sentence)
How do I use it? (copy-pasteable example)
What does the output look like? (sample JSON)

That's it. I rewrote all 77 READMEs to follow this format and engagement went up immediately.

4. Shipping beats perfecting

I published my first scraper with a bug that crashed on subreddits with special characters. A user reported it within an hour. I fixed it in 10 minutes.

If I'd waited to "make it perfect," I'd still be writing unit tests for edge cases nobody would ever hit.

5. The best marketing is showing up consistently

500+ articles might sound like a lot. Most of them have <20 views. But a few hit 50+ views, and those brought in actual leads.

Consistency compounds. The article I wrote 6 months ago still brings traffic today.

What I'd tell past-me

Start building before you feel ready
Pick boring technology
Ship something ugly, then iterate
Write about what you build
The money follows the value, not the other way around

I'm still building. You can see everything at Apify and GitHub.

What side project taught you the most? Not the one that made money — the one that made you a better developer.

Looking for a developer who ships fast? Email me.

DEV Community