My Python tests passed. Production still broke.

#python #testing #debugging #programming

My Python tests passed. Production still broke.

Tests all green. Pushed to prod. Got paged 20 minutes later.

Building this data pipeline that pulls customer orders from an API and processes them. Simple stuff. Fetch JSON, validate fields, insert to database. Wrote unit tests for validation logic. Everything passed.

def validate_order(order):
    if not order.get('customer_id'):
        raise ValueError('Missing customer_id')
    if not order.get('total'):
        raise ValueError('Missing total')
    if order['total'] <= 0:
        raise ValueError('Invalid total')
    return True

Test coverage was 100%. Felt good about it.

Prod threw errors on real orders

API sometimes returns total as a string instead of a number. My test mocks always used integers.

# This passed my tests
test_order = {'customer_id': 123, 'total': 50}

# This broke in production  
real_order = {'customer_id': 123, 'total': '50.00'}

The comparison order['total'] <= 0 worked fine with integers. With strings? Python just compared lexicographically. '50.00' > 0 evaluates to True because string comparison.

Didnt catch it because my test data was too clean.

Fixed it by adding type coercion:

def validate_order(order):
    if not order.get('customer_id'):
        raise ValueError('Missing customer_id')

    total = order.get('total')
    if not total:
        raise ValueError('Missing total')

    # Handle string or numeric total
    try:
        total = float(total)
    except (ValueError, TypeError):
        raise ValueError(f'Invalid total format: {total}')

    if total <= 0:
        raise ValueError('Total must be positive')

    return True

Also updated tests to use realistic data:

# Test with string values like real API
test_order = {'customer_id': '123', 'total': '50.00'}
validate_order(test_order)  # Now this actually tests the real scenario

Stopped mocking everything

Grabbing real API responses now and using those in test fixtures. Made a small script that hits the API and saves responses to tests/fixtures/real_responses.json. Run it weekly to catch when APIs change format.

# tests/fixtures/capture_real_data.py
import requests
import json

response = requests.get('https://api.example.com/orders?limit=10')
with open('real_responses.json', 'w') as f:
    json.dump(response.json(), f, indent=2)