The main difference between a developer and an engineer is not just the code they write. It's how they think about building a system. How they optimize. How they use resources.
I learned this from a real project.
I had to read 10,000 JSON requests and write them into a file. Simple enough. I wrote the program the way I always did the developer way. Read all 10,000 requests, store everything in memory, then write to the file. Done.
When I tested it, my memory usage was much higher than normal. Way higher than it should have been.
The developer in me would have just moved on. It works, right? But something made me stop and ask why is this consuming so much memory?
I dug into it. And I found the problem.
What I was doing was basically doubling my memory usage without realizing it. First, I was loading all 10,000 records into memory to store the data. Then, I was holding all of it again in memory while writing to the file. Two copies of the same data sitting in memory at the same time. That's the thing about this approach it's not reliable when you're dealing with a large number of requests. It doesn't scale. It just quietly eats your resources.
That's when I found streams.
The idea behind streaming is simple but powerful. Instead of loading everything at once, you break the data into small chunks. At any given moment, only one chunk lives in memory. You read it, transform it, write it and move on to the next one.
The transform step is the interesting part. It's not just about moving data from one place to another. Transform lets you validate each chunk, check if the structure is correct, clean it if needed before it ever reaches the file. So you're not just being efficient with memory, you're also processing your data with more control.
And because you're always working on one small piece at a time, the memory usage stays low and consistent no matter if you're processing 100 requests or 100,000.
That one question am I using my resources in a reliable way? is what pushed me from just solving the problem to actually understanding it. That's the engineer mindset. Not just making something work, but thinking about how it works under the hood and what happens when it scales.
That shift in thinking changes everything.
Top comments (0)