DEV Community: Maksim Pugin

Why Your Python Script Works Locally but Fails in Production

Maksim Pugin — Mon, 02 Nov 2026 06:54:43 +0000

Your code works perfectly on your machine. Then you deploy it, and everything starts breaking. Sound familiar?

There's a moment almost every Python developer experiences.

You've spent hours building a script. You've tested it several times. Everything works exactly as expected.

Then you move it to a server.

And suddenly:

The API stops responding.
A file can't be found.
A library behaves differently.
The application crashes without a useful error message.
A process that previously took 30 seconds now runs indefinitely.

The frustrating part?

You haven't changed the code.

So what went wrong?

The answer is usually simple: your local environment and your production environment are not the same.

And writing code that works is only the first step toward building software that works reliably.

Let's explore seven common problems that turn working Python scripts into production failures — and how to prevent them.

1. The Hidden Danger of Missing Timeouts

Let's start with something that looks completely harmless.

import requests

response = requests.get(
    "https://api.example.com/products"
)

print(response.json())

This code works perfectly when the API is available and responds quickly.

But what happens when the server stops responding?

By default, the requests library doesn't set a timeout.

That means your application could wait indefinitely for a response.

Now imagine this script is running as part of a scheduled job.

One unavailable API could block the entire workflow.

A better approach

import requests

def fetch_products():
    try:
        response = requests.get(
            "https://api.example.com/products",
            timeout=(3, 10)
        )

        response.raise_for_status()

        return response.json()

    except requests.Timeout:
        print("Request timed out")

    except requests.RequestException as error:
        print(f"Request failed: {error}")

    except ValueError:
        print("Invalid JSON response")

    return None

Here, the timeout configuration specifies:

3 seconds for establishing the connection.
10 seconds for waiting between socket read operations.

One important detail: these timeouts are not a strict 13-second limit on the total request duration.

A server that continuously sends small amounts of data may keep the connection active much longer.

For strict execution deadlines, you need additional controls at the application or infrastructure level.

Lesson: External services are not guaranteed to respond, even when your code is correct.

2. Hardcoded Paths Work Until They Don't

Consider a script that saves data to a CSV file.

import csv

with open(
    "C:/Users/Developer/Desktop/products.csv",
    "w"
) as file:
    writer = csv.writer(file)
    writer.writerow(["id", "name", "price"])

This might work perfectly on your Windows laptop.

But what happens when you deploy it to a Linux server?

The directory doesn't exist.

The application fails.

This problem becomes even more common when working with Docker containers, virtual machines, or cloud environments.

Use pathlib instead

Python provides a much better way to handle filesystem paths.

from pathlib import Path
import csv

BASE_DIR = Path(__file__).resolve().parent

DATA_DIR = BASE_DIR / "data"
DATA_DIR.mkdir(
    parents=True,
    exist_ok=True
)

OUTPUT_FILE = DATA_DIR / "products.csv"

with OUTPUT_FILE.open(
    "w",
    newline="",
    encoding="utf-8"
) as file:
    writer = csv.writer(file)
    writer.writerow(["id", "name", "price"])

This approach is more portable and avoids operating-system-specific path separators.

However, there's another important production consideration.

The application directory might be read-only.

In containerized environments, you may also lose local files when a container is replaced.

For persistent data, configure an appropriate storage location or use an external storage service.

Lesson: Never assume that the production filesystem looks like your development machine.

3. Your Dependencies Are Part of Your Application

Here's a situation that can be surprisingly difficult to debug.

You develop a Python application using several libraries.

Everything works.

A colleague clones the repository and runs the same code.

It crashes.

Why?

Because your environments contain different dependency versions.

For example, your application might rely on behavior that changed between major versions of a library.

The wrong approach

pip install requests beautifulsoup4 pandas

This installs available versions that satisfy the requirements at installation time.

But those versions may differ across machines or deployments.

A more reliable approach

For a simple project, you can maintain a dependency file with pinned versions.

requests==2.32.3
beautifulsoup4==4.12.3
pandas==2.2.3

These are example versions, not necessarily the versions your project should use.

Then install dependencies with:

python -m pip install -r requirements.txt

For larger applications, consider tools such as uv, Poetry, or pip-tools to generate and maintain lockfiles.

Also remember that dependency versions aren't the entire environment.

Python version, operating system, native libraries, and system packages can affect application behavior.

For example, a package with native extensions might work on one platform but fail on another.

Lesson: Reproducibility is a requirement, not a convenience.

4. Print Statements Are Not a Logging Strategy

During development, many of us debug code like this:

print("Starting parser")
print("Opening page")
print("Extracting products")
print("Done")

There's nothing inherently wrong with print().

For small scripts, it's often sufficient.

But imagine your application runs every 15 minutes on a remote server.

One execution fails at 3:00 AM.

You need to know:

When did the failure occur?

Which operation failed?

What exception was raised?

How frequently has this happened?

Was the failure temporary or persistent?

Random print statements quickly become difficult to manage.

Use Python's logging module

import logging

logging.basicConfig(
level=logging.INFO,
format=(
"%(asctime)s | "
"%(levelname)s | "
"%(name)s | "
"%(message)s"
)
)

logger = logging.getLogger(name)

def process_products():
logger.info("Starting product processing")

try:
    result = 10 / 0
    return result

except ZeroDivisionError:
    logger.exception(
        "Product processing failed"
    )
    raise

The logger.exception() method automatically includes traceback information when called inside an exception handler.

That makes debugging significantly easier.

For production systems, structured JSON logs can make filtering and analyzing events even more effective.

One important rule: never log passwords, access tokens, or other sensitive information.

Lesson: If your application fails and you can't explain why, you have an observability problem.

Environment Variables Are Better Than Hardcoded Secrets

Imagine finding this in a repository:

DATABASE_HOST = "192.168.1.100"
DATABASE_USER = "admin"
DATABASE_PASSWORD = "super_secret_password"

There are two problems.

First, these settings might only work in one environment.

Second, credentials embedded in source code create unnecessary security risks.

Separate configuration from code

import os

DATABASE_HOST = os.environ.get(
"DATABASE_HOST",
"localhost"
)

DATABASE_USER = os.environ["DATABASE_USER"]

DATABASE_PASSWORD = os.environ[
"DATABASE_PASSWORD"
]

Now the application can use different settings in development, testing, and production without modifying its source code.

For example:

export DATABASE_HOST="db.internal"
export DATABASE_USER="app_user"
export DATABASE_PASSWORD="your-secret"

Environment variables are useful, but they aren't automatically a complete secrets-management solution.

For sensitive production credentials, consider using a dedicated secrets manager, restricting access, and rotating credentials regularly.

Lesson: Configuration should change between environments without requiring code changes.

Successful Execution Doesn't Mean Correct Results

This is one of the most dangerous problems in data-processing applications.

Imagine a web scraper that collects product information.

The script runs successfully.

No exceptions occur.

The output file is created.

Everything looks fine.

Except the file contains only 8 products instead of the expected 200.

Technically, the program completed.

But from a business perspective, it failed.

This is especially common in web scraping, ETL pipelines, API integrations, and scheduled data-processing jobs.

Validate the output

def validate_products(products):
if not isinstance(products, list):
raise TypeError(
"Expected a list of products"
)

if not products:
    raise ValueError(
        "No products were collected"
    )

for index, product in enumerate(products):
    if not isinstance(product, dict):
        raise TypeError(
            f"Invalid product at index {index}"
        )

    required_fields = {
        "id",
        "name",
        "price"
    }

    missing = required_fields - product.keys()

    if missing:
        raise ValueError(
            f"Product {index} is missing: {missing}"
        )

return True

This example checks the basic structure of the collected data.

In a real application, you might also validate:

Field types and allowed values.

Duplicate records.

Unexpectedly low record counts.

Missing identifiers.

Data freshness.

Sudden changes in price distributions.

A nonempty list is not proof that a scraping job succeeded.

And a zero exit code is not proof that the business requirements were satisfied.

Lesson: Validate outcomes, not just execution.

Your Script Needs a Recovery Strategy

Let's imagine a Python application processing 10,000 products.

After processing 8,500 products, the server restarts.

What happens next?

If the application starts from the beginning, it might repeat thousands of operations.

If it doesn't save progress, some data could be lost.

This is where a simple script starts becoming a reliable system.

Introduce checkpoints

A basic approach is to track completed product identifiers.

import json
from pathlib import Path

CHECKPOINT_FILE = Path(
"completed_products.json"
)

def load_completed():
if not CHECKPOINT_FILE.exists():
return set()

with CHECKPOINT_FILE.open(
    "r",
    encoding="utf-8"
) as file:
    return set(json.load(file))

def save_completed(completed):
temporary = CHECKPOINT_FILE.with_suffix(
".tmp"
)

with temporary.open(
    "w",
    encoding="utf-8"
) as file:
    json.dump(
        sorted(completed),
        file
    )

temporary.replace(CHECKPOINT_FILE)

Now a processing loop can skip previously completed records.

completed = load_completed()

for product_id in product_ids:
if product_id in completed:
continue

process_product(product_id)

completed.add(product_id)
save_completed(completed)

This is a useful starting point for a single-process script.

However, it has limitations.

It assumes process_product() completes its work before the checkpoint is saved. If a crash occurs between an external side effect and the checkpoint update, that operation may be repeated.

For this reason, production workflows should consider idempotency.

An idempotent operation can be safely repeated without producing unwanted additional effects.

For larger systems, database-backed job states, durable queues, and transactional processing are often more appropriate than local JSON files.

Lesson: A reliable application should be designed for interruptions, not just successful runs.

A Practical Production Readiness Checklist

Before deploying a Python application, I find it useful to ask a few questions.

Reliability

Do external requests have appropriate timeouts?

Are expected failures handled explicitly?

Can interrupted operations be safely retried?

Does the application validate its output?

Configuration

Are configuration values separated from source code?

Are secrets excluded from the repository?

Are dependency versions reproducible?

Does the application work in the target environment?

Observability

Are important operations logged?

Can failures be diagnosed from logs?

Are sensitive values excluded from logs?

Is there a way to detect repeated failures?

Data integrity

Can the application recover after a restart?

Are duplicate operations handled safely?

Is persistent data stored appropriately?

Can incomplete results be detected?

You don't need a complicated microservices architecture to address these problems.

Even a small Python application benefits from these principles.

The Difference Between a Script and a System

One of the most valuable lessons in software development is understanding that a working implementation is not necessarily a reliable implementation.

A script answers the question:

"Can this task be automated?"

A production-ready system must answer additional questions:

"What happens when something goes wrong?"

"Can we detect the problem?"

"Can we recover without losing data?"

"Can another developer maintain this six months later?"

These questions fundamentally change how we design software.

The code itself might represent only a small portion of the engineering effort.

The rest involves understanding failure scenarios, infrastructure, dependencies, observability, and operational requirements.

And that's why moving from a local Python script to a production environment is such an important learning experience.

Final Thoughts

Python makes it incredibly easy to build useful applications quickly.

That's one of its greatest strengths.

But the simplicity of writing Python code can sometimes hide the complexity of operating software in real environments.

A script that works on your laptop is a good beginning.

A system that continues working when networks fail, servers restart, dependencies change, and unexpected data appears is a much more significant achievement.

The difference between a working script and reliable software isn't the number of lines of code. It's how well you've prepared for everything that can go wrong.

And that mindset is what turns programming into software engineering.

What's the most unexpected problem you've encountered when deploying a Python application?

Was it an environment issue, an API failure, a dependency conflict, or something completely different?

Share your experience in the comments. I'd love to hear about it.

AI Makes Coding Faster. It Doesn't Necessarily Make Software Better.

Maksim Pugin — Tue, 20 Oct 2026 08:45:13 +0000

We're getting better at generating code. But are we getting better at building software?

A few years ago, writing software meant spending a significant amount of time searching documentation, debugging syntax errors, and implementing repetitive logic.

Today, AI can generate an API endpoint, write a database query, explain an unfamiliar codebase, or produce a working prototype in seconds.

That's impressive.

But there's a question I think deserves more attention:

If writing code is becoming easier, why is building reliable software still so difficult?

The answer is that generating code and engineering software are two different problems.

And AI is changing one much faster than the other.

1. The Illusion of Productivity

Imagine two developers working on the same application.

Developer A writes most of the code manually. They spend time planning the architecture, defining interfaces, and thinking through potential failures.

Developer B uses AI to generate the majority of the implementation.

After two hours, Developer B has significantly more code.

But does that mean the project is further along?

Not necessarily.

Lines of code are not a meaningful measure of software quality.

What matters is whether the application is correct, maintainable, secure, and capable of handling real-world conditions.

AI can dramatically reduce implementation time while leaving many engineering decisions unresolved.

The dangerous assumption is that faster implementation automatically means faster delivery.

Sometimes it does.

Sometimes it simply moves the complexity into debugging, integration, and maintenance.

2. A Working Prototype Is Not a Reliable System

Consider a simple Python web scraper.

You ask an AI assistant:

"Create a Python script that extracts product names and prices from an e-commerce website."

Within seconds, you receive something that looks reasonable.

import requests
from bs4 import BeautifulSoup

response = requests.get(
    "https://example.com/products",
    timeout=10
)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")

for product in soup.select(".product"):
    name = product.select_one(".name")
    price = product.select_one(".price")

    if name and price:
        print(name.get_text(strip=True))
        print(price.get_text(strip=True))

For a simple static page with matching selectors, this might work.

But now consider a real marketplace.

What happens when:

Products are loaded dynamically?
Prices change based on customer location?
Some products have multiple price types?
A request times out halfway through execution?
The HTML structure changes?
The same product appears multiple times?
The scraper returns incomplete data without raising an exception?

The initial script doesn't address these problems.

And that is not necessarily a failure of AI.

It's a failure to define the actual engineering requirements.

AI can generate an implementation of the problem you describe. It cannot guarantee that you've described the right problem.

3. Architecture Is Where the Real Work Begins

While developing a Python application for collecting marketplace data, I encountered a challenge that goes beyond HTML parsing.

The application wasn't just a scraper.

It involved several connected components:

Browser automation with Selenium.
HTML parsing with BeautifulSoup.
Data validation and processing.
CSV and JSON exports.
User authentication and MySQL storage.
A desktop interface built with Tkinter.

Each component was manageable individually.

The difficulty came from making them work together reliably.

For example, browser automation can take time. If it runs directly in the graphical interface's main thread, the application may become unresponsive.

Moving that operation into a background thread improves responsiveness, but introduces additional questions.

How should results be communicated to the interface?

What happens when a user cancels an operation?

How should partial results be handled?

How do we prevent concurrent tasks from corrupting shared state?

These are architectural decisions.

AI can suggest solutions, but the developer still needs to evaluate the trade-offs.

A useful design principle is to separate responsibilities:

User Interface
      |
      v
Application Controller
      |
      v
Scraping Service
      |
      v
Data Validation
      |
      v
Storage / Export

This structure isn't automatically the best architecture for every project.

But it creates boundaries that make the application easier to reason about.

And reasoning about a system is something developers cannot afford to outsource completely.

4. The Problem with Code You Don't Understand

One of the most interesting effects of AI-assisted development is how easy it becomes to accept code that looks correct.

A generated function may have:

Clean formatting.
Descriptive variable names.
Type annotations.
Helpful comments.
A convincing explanation.

None of these guarantee correctness.

Consider a function that calculates a product discount:

def discount_percent(original, current):
    return round(
        (original - current) / original * 100,
        2
    )

Looks fine, right?

But what happens if original is zero?

What if one of the prices is missing?

What if the price is represented as a localized string?

What if the current price is higher than the original price?

The function solves the simplest version of the problem.

A reliable implementation requires explicit decisions about invalid input, numerical precision, and business rules.

The same principle applies to AI-generated database queries, API integrations, authentication logic, and infrastructure configuration.

Code that looks professional is not necessarily production-ready code.

5. AI Changes the Developer's Responsibility

I don't think the solution is to avoid AI.

Quite the opposite.

AI is useful for:

Exploring unfamiliar libraries.
Generating boilerplate.
Suggesting refactoring approaches.
Creating initial test cases.
Explaining errors.
Reviewing alternative implementations.
Accelerating documentation.

But I believe its greatest value appears when developers treat it as an engineering assistant rather than an unquestionable authority.

A more reliable workflow looks like this:

Step 1: Define the problem.

Clarify inputs, outputs, constraints, and expected behavior.

Step 2: Design the architecture.

Decide which components are needed and how they interact.

Step 3: Use AI to accelerate implementation.

Generate focused components rather than blindly producing an entire application.

Step 4: Review the result.

Understand the generated code and challenge its assumptions.

Step 5: Test real failure scenarios.

Validate edge cases, integration behavior, and error handling.

Step 6: Measure the outcome.

Evaluate correctness, performance, maintainability, and actual time saved.

This workflow may feel slower than generating an entire application from one prompt.

But it can prevent expensive problems later.

6. The New Engineering Bottleneck

Here's the interesting part.

As code generation becomes cheaper, the relative importance of other engineering activities increases.

Understanding requirements.

Designing system boundaries.

Validating behavior.

Managing complexity.

Making trade-offs.

Debugging unexpected interactions.

These activities were always important.

AI simply makes the difference between writing code and engineering software more visible.

The bottleneck is gradually shifting from implementation toward judgment and verification.

This also changes what developers need to learn.

Knowing syntax remains useful.

But understanding how systems behave becomes even more valuable.

7. My Prediction: Software Engineering Will Become More About Decisions

I don't believe AI will make software engineering irrelevant.

I believe it will change which engineering skills matter most.

Developers who only translate detailed instructions into code may face increasing automation.

Developers who can understand complex problems, design reliable systems, evaluate generated solutions, and take responsibility for outcomes will remain valuable.

The future of software development isn't necessarily about writing every line manually.

It's about knowing which lines should exist, why they exist, and whether they actually solve the problem.

Final Thoughts

AI has made it easier than ever to create something that works.

But creating something that works once is not the same as creating something that works reliably.

The real measure of engineering productivity isn't how quickly we generate code. It's how effectively we turn requirements into dependable software.

I'm interested in how other developers are experiencing this shift.

Has AI actually reduced the total time you spend delivering software, or has it moved more of your time into reviewing and debugging generated code?

I'd love to hear your experience.

Building a Web Scraper with Python: Lessons from a Real Project

Maksim Pugin — Tue, 20 Oct 2026 08:23:09 +0000

Building a Web Scraper with Python: Lessons from a Real Project

When I started developing my own web scraping application, I thought the hardest part would be extracting data from websites.

I was wrong.

Extracting information is relatively straightforward. Building a system that can consistently collect, process, store, and present that information is a completely different challenge.

In this article, I'd like to share some lessons from developing a Python-based desktop application for collecting and analyzing marketplace product data.

The Idea Behind the Project

The initial idea was simple: create an application that could automatically collect product information from online marketplaces and present it in a structured format.

The application needed to handle several tasks:

Search for products based on user queries.
Collect product names, prices, discounts, and ratings.
Process and structure the extracted information.
Export results to CSV and JSON.
Provide a graphical interface for interacting with the collected data.

However, as development progressed, the project became much more interesting from an engineering perspective.

Choosing the Technology Stack

I decided to build the application using Python and several libraries:

Python — the main programming language.

Selenium — browser automation and interaction with dynamically rendered pages.

BeautifulSoup — parsing HTML and extracting structured information.

MySQL — storing user accounts and profile information.

Tkinter and ttkbootstrap — creating the desktop interface.

Pandas and CSV/JSON tools — potential options for further data processing, with CSV and JSON already used for exporting results.

Each technology solves a specific problem, but the real challenge is making all these components work together.

Challenge #1: Modern Websites Are Dynamic

One of the first problems I encountered was that many marketplace pages don't expose all their product information in the initial HTML response.

Content may be loaded dynamically as users scroll or interact with the page.

This means that simply downloading the HTML using an HTTP request is not always enough.

My approach involved using Selenium to interact with the browser, scrolling through product listings, collecting product URLs, and then extracting information from individual product pages.

The key lesson?

Web scraping is often about understanding how a website behaves, not just how its HTML is structured.

Challenge #2: Data Is Rarely Consistent

Another interesting challenge was processing product information.

For example, a product might have:

A regular price.
A discounted price.
A special price available through a marketplace payment method.
A discount percentage.
A rating and review count.

Not every product contains every field, and the HTML structure can vary.

To handle this, I had to implement additional validation and data-cleaning logic.

A scraper that extracts information is useful.

A scraper that extracts reliable and consistently structured information is significantly more valuable.

Challenge #3: Architecture Matters

At first, it may seem reasonable to put the entire scraping process into a single script.

But once the application includes authentication, database operations, browser automation, data processing, and a graphical interface, this approach becomes difficult to maintain.

I started separating responsibilities into logical components:

User authentication and profile management.
Browser automation and URL collection.
Product information extraction.
Data processing and export.
Graphical user interface.

This made the project easier to understand, debug, and extend.

I also used background threads to prevent long-running scraping operations from freezing the interface.

One important consideration is keeping GUI updates on the main Tkinter thread, while passing results from background operations safely.

Challenge #4: Reliability Is More Important Than Speed

One of the most important things I learned is that collecting data quickly is not necessarily the same as collecting it successfully.

A scraping application needs to account for:

Network failures.
Unexpected HTML changes.
Missing product information.
Browser timeouts.
Duplicate records.
Partial execution failures.

A production-oriented approach should also respect website policies, reasonable request rates, and access restrictions.

These considerations are essential if the application is expected to operate reliably over time.

What's Next?

I'm interested in taking this project further.

My long-term goal is to move beyond simple data collection and develop a system capable of analyzing marketplace information and identifying useful patterns.

Potential development directions include:

Historical price tracking.
Product comparison across marketplaces.
Automated detection of price changes.
AI-assisted product classification.
Recommendation algorithms.
A more scalable data-processing architecture.

The most interesting part is the transition from a basic scraping tool to a complete data-driven product.

Final Thoughts

This project reminded me of something important:

Writing code that works is only the first step. Designing software that remains understandable, maintainable, and useful is the real engineering challenge.

I'm continuing to explore Python, automation, AI, and enterprise software development, and I plan to share more practical engineering experiences here.

If you've worked on web scraping or data collection systems, I'd be interested to hear:

What was the biggest technical challenge you encountered when building a scraper?

Thanks for reading!