DEV Community: Satish Kandala

Your Python Script Works on 10K Rows. What Happens at 10 Million?

Satish Kandala — Mon, 17 Aug 2026 10:34:00 +0000

Python script can look perfectly efficient when you test it with 10,000 records.

Then production sends 10 million.

Suddenly:

memory consumption increases dramatically
processing becomes slower
one bad record can interrupt an entire run
database and API calls become bottlenecks
code that looked harmless during development starts behaving very differently
This is an important transition for Data Engineers.

The question is no longer:

Does the Python code work?

The better question becomes:

Will the same design still work when the data becomes much larger?

Let's look at five Python patterns that become increasingly important as data volume grows.

Be Careful About Loading Everything Into Memory Consider this simple transformation:

processed_records = [
transform(record)
for record in records
]
There is nothing inherently wrong with this code.

For a relatively small dataset, it may be exactly what you need.

The problem appears when records becomes very large.

A list comprehension builds the resulting list in memory. If millions of transformed objects are produced, memory usage can increase quickly.

One alternative is a generator expression:

processed_records = (
transform(record)
for record in records
)
The difference looks tiny.

The execution model is not.

A generator produces values lazily instead of constructing the complete result immediately.

For example:

def transformed_records(records):
for record in records:
yield transform(record)
Now we can process records progressively:

for record in transformed_records(records):
write_to_destination(record)
This leads to an important Data Engineering principle:

Don't keep data in memory longer than necessary.

Generators are not automatically faster, and they are not the solution to every performance problem.

But when the workflow is naturally sequential, lazy evaluation can significantly reduce memory pressure.

Don't Read a Large File All at Once Unless You Need To This pattern is convenient:

with open("customers.csv") as file:
rows = file.readlines()
But readlines() loads the entire file into memory.

For a small file, nobody cares.

For a multi-gigabyte file, the decision becomes much more important.

Python file objects are iterable, so we can process a file progressively:

with open("customers.csv") as file:
for row in file:
process(row)
Now the application doesn't need the entire file available in memory simultaneously.

A slightly more realistic version might be:

import csv

with open("customers.csv", newline="") as file:
reader = csv.DictReader(file)

for record in reader:
    transformed = transform(record)
    write_to_destination(transformed)

This pattern works particularly well when the pipeline is:

Read → Transform → Write

and each record can be processed independently.

The important lesson is not simply:

"readlines() is bad."

It isn't.

The real lesson is:

Understand the memory implications of the operation you're choosing.

Sometimes Batch Processing Is Better Than Row-by-Row Processing Processing one record at a time reduces memory usage.

But it can create another problem.

Imagine inserting one million records into a database individually.

for record in records:
insert_into_database(record)
Even if the Python processing is efficient, one million database round trips can make the pipeline painfully slow.

This is where batching becomes useful.

A simple batch generator could look like this:

def create_batches(items, batch_size):
batch = []

for item in items:
    batch.append(item)

    if len(batch) == batch_size:
        yield batch
        batch = []

if batch:
    yield batch

Now:

for batch in create_batches(records, 1000):
insert_batch(batch)
Instead of:

1,000,000 individual writes

we potentially perform:

1,000 writes containing 1,000 records each.

The exact numbers depend on the system, but the principle is extremely useful.

Batching is common when working with:

database inserts
REST APIs
message queues
object storage
ETL transformations
external services
However, larger batches are not automatically better.

Very large batches may:

consume more memory
increase transaction size
increase retry cost
cause API payload limits
create longer-running database locks
Good Data Engineering often means finding the correct balance.

At Scale, Failure Handling Becomes Part of the Design Suppose we process 10 million records and record number 7,452,819 contains invalid data.

Should the entire pipeline fail?

Sometimes yes.

Sometimes absolutely not.

This code is dangerous:

for record in records:
process(record)
because one unexpected exception can terminate the complete loop.

A better design might be:

import logging

logger = logging.getLogger(name)

for record in records:
try:
process(record)

except ValueError as error:
    logger.warning(
        "Invalid record %s: %s",
        record.get("id"),
        error,
    )

But even this is only the beginning.

Production pipelines need a clear failure strategy.

For every failure, ask:

Should we retry it?
Useful for temporary network or service failures.

Should we skip it?
Possibly appropriate for non-critical malformed records.

Should we quarantine it?
Store invalid records separately so they can be investigated later.

Should the pipeline fail completely?
Sometimes data integrity is more important than availability.

For example, silently skipping invalid financial transactions could be much worse than stopping the pipeline.

So good exception handling is not:

except Exception:
pass
That hides problems.

Instead, failures should provide enough context to understand:

what failed → why it failed → what happened to the data → whether processing continued

Know When Plain Python Is No Longer the Right Tool This is probably the most important point.

Suppose you've already improved:

memory usage
file processing
batching
database operations
error handling
But the workload continues growing.

10 million rows becomes 100 million.

Then 500 million.

Eventually the question changes.

It is no longer:

How can I optimize this Python loop?

It becomes:

Should this workload still run inside one Python process on one machine?

That is where distributed processing systems become relevant.

A tool such as PySpark can divide a dataset into partitions and process those partitions across multiple executors.

Conceptually:

Large Dataset

↓

Partition 1 → Executor 1
Partition 2 → Executor 2
Partition 3 → Executor 3
Partition 4 → Executor 4

↓

Combined Result
Instead of asking one process to do all the work, we distribute the workload.

But this does not mean:

Big dataset = always use Spark.

Distributed computing introduces its own costs:

cluster infrastructure
serialization
network communication
data shuffling
partition management
scheduling overhead
operational complexity
For some workloads, well-written Python running on one machine is more than sufficient.

For others, distributed processing becomes necessary.

Knowing when to make that transition is an important Data Engineering skill.

A Simple Experiment You Can Try
Instead of only reading about these patterns, test them.

Generate a large dataset:

records = range(10_000_000)
Then compare several approaches.

Approach 1 — Build a List
results = [
value * 2
for value in records
]
Approach 2 — Use a Generator
results = (
value * 2
for value in records
)
Then consume it:

for value in results:
pass
Observe:

memory consumption
execution time
when computation occurs
how iteration behaves
Next, experiment with batching.

for batch in create_batches(records, 1000):
process_batch(batch)
Try:

100
1,000
10,000
100,000
as different batch sizes.

Don't just ask which one is fastest.

Ask:

Why does the behaviour change?

That question usually teaches more than the benchmark itself.

Scaling Is Usually About Trade-offs
One of the biggest lessons I've learned from Data Engineering is that performance problems rarely have one universal solution.

You might optimize memory and increase CPU usage.

You might increase batch size and increase retry cost.

You might introduce parallelism and increase complexity.

You might move to Spark and discover that the workload was too small to justify a distributed engine.

Engineering is about understanding those trade-offs.

The same code can be:

perfectly acceptable for 10K rows

and

completely inappropriate for 100 million rows.

Context matters.

The Mental Model I Use
When a Python data-processing workload starts growing, I think about it in roughly this order:

Can I avoid unnecessary work?

↓

Can I process data lazily?

↓

Can I stream instead of loading everything?

↓

Can I batch expensive operations?

↓

Can I reduce I/O or database round trips?

↓

Can I parallelize safely?

↓

Does the workload now justify
distributed processing?
Jumping directly to Spark isn't always the solution.

But trying to force every workload through a single Python process isn't the solution either.

The important skill is recognizing where that boundary lies.

Final Thought
Writing Python that produces the correct output is the first step.

Writing Python that remains reliable when:

data grows, failures occur, memory becomes constrained, external systems slow down, and requirements change

is a different level of engineering.

So the next time a Python pipeline works perfectly with 10,000 records, try asking:

What happens if tomorrow this becomes 10 million?

That one question can completely change the way you design the solution.

Challenge
Take one of your existing Python data-processing scripts and answer these five questions:

Does it load the complete dataset into memory?
Could any operation be processed lazily?
Are database/API operations executed individually when they could be batched?
What happens when one record fails?
At what data volume would you consider moving the workload to a distributed processing engine?
I'd be interested to hear how you approach the Python → distributed processing transition in real Data Engineering projects.

Master SQL Queries with Interactive Practice

Satish Kandala — Sat, 14 Mar 2026 06:08:01 +0000

If you're looking to improve your SQL skills, practicing with real-world queries is essential. Whether you're preparing for technical interviews or building stronger database skills for your day job, hands-on practice makes all the difference.

I've built an interactive SQL practice platform that helps you master queries at your own pace. It features a collection of carefully designed SQL problems ranging from basic SELECT statements to complex JOIN operations and aggregate functions.

What makes this platform effective:

Real-world database scenarios
Instant feedback on your queries
Progressive difficulty levels
Practice problems across different SQL topics

Check it out at https://www.sql-practice.online/ and start practicing today. The platform is free to use and perfect for anyone wanting to solidify their SQL fundamentals.

Whether you're a beginner just starting with databases or looking to refine your advanced SQL knowledge, there's something here for everyone. Happy querying!

Master Python, PySpark, and SQL with Real-World Practice

Satish Kandala — Sat, 14 Mar 2026 05:47:15 +0000

Are you learning Python, PySpark, or SQL? Whether you're a student or an experienced developer, practicing with real-world scenarios is crucial.

We've built an interactive learning platform that helps you master these technologies through hands-on practice. No boring lectures—just interactive coding exercises that mimic real engineering challenges.

Key Features:

Interactive Practice: Write and execute code directly in your browser. Get instant feedback on your solutions.

Real-World Use Cases: Learn through examples pulled from actual data engineering and backend development scenarios.

SQL Mastery: From basic queries to complex joins and window functions.

Python Fundamentals to Advanced: Learn data structures, OOP, and functional programming patterns.

PySpark for Data: Master distributed computing with PySpark for big data processing.

Structured Learning Paths: Progressive difficulty levels designed for beginners to intermediate developers.

Whether you're preparing for technical interviews, building skills for your next role, or just passionate about data engineering, we've got you covered.

Check it out: www.py-spark-sql.com/

Your feedback and suggestions are welcome. Let's build better developers together.

Company-Level SQL Learning Mode Now Available on SQL Practice Online

Satish Kandala — Mon, 22 Dec 2025 13:11:43 +0000

Introducing Company-Level SQL Learning Mode

We're thrilled to announce a major update to SQL Practice Online: Company-Level Learning Mode is now available!

What is Company-Level Learning Mode?

Company-Level Learning Mode is a structured approach to SQL learning that mirrors real-world technical interviews and on-the-job scenarios. Instead of practicing random SQL problems, learners can now focus on SQL question sets specifically curated for major tech companies like:

Google
Amazon
Meta
Microsoft
Apple
And more...

Why Company-Level Mode?

1. Targeted Preparation

Each company has unique SQL problem patterns. Our company-level mode focuses on the exact types of questions you'll encounter in interviews or technical assessments at that specific company.

2. Progressive Difficulty

Questions are organized by difficulty level within each company's problem set, allowing you to build confidence gradually.

3. Real Interview Patterns

Our questions are modeled after actual technical interview problems, giving you authentic practice that translates directly to real opportunities.

4. Track Your Progress

Monitor your performance across different companies and skill levels. Identify weak areas and focus your learning where it matters most.

How to Get Started

Visit sql-practice.online
Select "Company-Level Mode" from the practice options
Choose your target company
Start practicing with problems curated specifically for that company

Sample Companies Available

Our Company-Level Mode currently features SQL questions for:

Google - Focus on window functions and complex joins
Amazon - Heavy emphasis on data aggregation and optimization
Meta - SQL problems with user engagement and analytics focus
Microsoft - Queries involving CTEs and advanced SQL concepts

Perfect For

✅ Job seekers preparing for technical interviews
✅ Developers wanting to strengthen SQL fundamentals
✅ Data engineers brushing up on SQL skills
✅ Anyone seeking structured, company-specific SQL practice

Join the SQL Practice Community

With over 50,000 monthly visitors, SQL Practice Online is the go-to platform for interactive SQL learning. Our community-driven approach means problems are constantly updated based on user feedback and real interview experiences.

Ready to ace your SQL interviews? Try Company-Level Learning Mode today at sql-practice.online and start preparing for your dream job!

Have questions or suggestions? Share your feedback in the comments below!

Master SQL with Interactive Practice: Free Platform for Beginners and Interviews

Satish Kandala — Thu, 18 Dec 2025 06:44:25 +0000

Learning SQL is essential for any developer or data professional, but tutorials and passive learning only get you so far. What you need is hands-on practice with real-world scenarios.

We built sql-practice.online to solve this problem. It's a free, web-based SQL learning platform designed for everyone from complete beginners to experienced developers preparing for interviews.

Key Features:

Interactive practice with instant feedback on every query
Real business datasets including HR, E-commerce, and School databases
Multiple difficulty levels from easy to advanced
Scenarios mirroring actual interview questions
No installation required - practice directly in your browser
Comprehensive coverage of SQL fundamentals to advanced techniques

Why This Matters:

SQL interviews aren't just about knowing syntax. Employers want to see that you can write efficient queries, understand data relationships, and solve real business problems. Our platform gives you exactly that experience.

Whether you're brushing up before interviews or building your SQL skills from scratch, sql-practice.online provides the guided, practical approach that actually works.

Start practicing for free at https://www.sql-practice.online/

Have you found interactive practice helpful in learning SQL? What features matter most to you? Share your thoughts in the comments.

Master SQL with Interactive Hands-On Practice

Satish Kandala — Fri, 28 Nov 2025 09:33:49 +0000

Why SQL Matters for Developers

SQL is one of the most essential skills for backend developers, data engineers, and software engineers. Yet many developers struggle with SQL fundamentals or don't get enough hands-on practice to master query optimization and complex data retrieval.

If you're preparing for technical interviews, building data-driven applications, or just want to strengthen your database skills, you need a platform that offers real-time, interactive practice.

Introducing SQL Practice Online

I'm excited to share sql-practice.online — a free, web-based SQL learning platform designed specifically for developers and engineering students who want to master SQL through interactive practice.

What Makes It Different

Hands-on Coding: Write and execute real SQL queries directly in your browser
Progressive Difficulty: Start with basics and advance to complex joins, subqueries, and optimization techniques
Instant Feedback: Get immediate results and explanations for your queries
Interview Prep: Practice real-world scenarios and common interview questions
Free & Open: No paywalls, no subscriptions — just pure learning

Perfect For

Engineering students preparing for technical interviews
Backend developers looking to strengthen SQL fundamentals
Data engineers working with databases and data pipelines
Anyone building or optimizing database-driven applications

Get Started Today

Visit https://www.sql-practice.online/ and start practicing SQL queries right now. Whether you're a beginner or experienced developer, you'll find challenges and practice problems that match your level.

Share Your Feedback

Have suggestions or want to see more features? Drop a comment below — I'd love to hear from the dev community about what would help you master SQL faster.

Happy coding! 🚀

Free Interactive SQL Practice Platform for Developers

Satish Kandala — Fri, 28 Nov 2025 06:09:24 +0000

Looking for a practical way to improve your SQL skills? Check out https://www.sql-practice.online/ — a free platform with 60+ interactive SQL exercises based on realistic business data (HR, E-commerce, School). Write and validate queries online, track your progress, and prepare for technical interviews. Perfect for learners, developers, and anyone who wants to master SQL through hands-on practice. No signup needed.