DEV Community: Mangabo Kolawole

I Spent Months Building a Financial Reconciliation Engine That Processes 10 Million Records in Under Five Minutes

Mangabo Kolawole — Sat, 18 Jul 2026 02:42:41 +0000

A few months ago I decided to build a reusable engine to make financial reconciliation easier. I had already done reconciliation work for companies in West Africa, closing out exports of around 100,000 transactions, and a lot of people in my network work in fintech. The one problem that keeps coming up in those conversations is reconciliation, and how painful it becomes once the data load grows.

So my first move was not to write code. It was to look for a tool that already solved the problem. To my surprise, there was almost nothing usable. The tools I found were either too general, because data reconciliation is a very broad problem, or they targeted finance but had not been maintained in years. Nothing was directed specifically at financial reconciliation and optimized for it. So I decided to take up the challenge. The problem was interesting on its own, and it was also a good way to deepen my knowledge of Go. The result is an open-source engine called Reconify, that can be used as a CLI, and you can find the repository here.

Here is where it landed. The current version reconciles 10 million records per side, 20 million in total, in 95 seconds on an eight-core Apple M1 Pro with 16 GB of RAM. It did not start there. The first design was fast on convenient files and fell apart on realistic ones, so the architecture had to change three times. The rest of this article is how.

Why reconciliation matters

A payment, most of the time has to work out on two sides. Your own ledger says the customer paid, and the provider is supposed to send the money and report it back. Reconciliation is the job of proving that those two records describe the same event, and flagging every case where they do not fully agree.

When they do not agree, real money is missing, a payment is double counted, or a settlement is stuck. So a finance team cannot close the books until the two sides are tied up.

The mechanic is quite simple to state. One side is the left source, usually your internal ledger. The other is the right source, the provider or bank export. The engine pairs a left row with a right row on a shared key, normally a reference like ORD-2026-84921, then classifies the pair:

Matched: both sides agree on the reference and the amount.
Mismatched: the reference matches but a field such as the amount does not.
Unmatched: a row sits on one side with no counterpart on the other.

This is a simple problem on a clean file. It stays easy until the files get large, wide, and inconsistent.

What makes reconciliation hard

Reconciling two financial exports looks simple until the files become large, wide, and slightly inconsistent. A provider may call a field merchant_reference, an internal ledger may call it external_reference, and a settlement file may represent several payments as one group. The engine still has to decide which records correspond, classify differences, preserve duplicate semantics, and explain every result.

I believe a customer should be able to reconcile a large daily export on a small cloud machine without turning the run into an unpredictable memory event. The benchmark therefore measured more than row count. It recorded schema width, input bytes, runtime, peak resident memory, throughput, and result parity.

Why I chose Go

I chose Go deliberately. At the start I assumed most of my advantage would come from concurrency, which turned out not to be true, at least not early on. I also wanted a language I could iterate in quickly, because the engine would go through several rewrites before it was good.

I did weigh the alternative honestly. I do not know Rust, so I vibe coded the same small routine in both Go and Rust and compared them. The gain was only about 10 to 20 percent in Rust's favor. For a language I already know and can write quickly, that trade was easy to accept.

Three iterations, one benchmark

The first design was fast on narrow synthetic files. A realistic benchmark exposed its cost. The engine then moved through three architectural iterations:

Architecture 1, memory-indexed streaming matching, released before the partitioned backend.
Architecture 2, partitioned execution, released in v0.4.0.
Architecture 3, parallel partition processing, released in v0.4.1.

The sections below walk each iteration in order, from the memory-bound first design to that partitioned run, alongside the benchmark that forced every change.

How to measure

Three metrics measure the engine's performance. Each one answers a different operational question, and no single one is enough on its own.

Runtime is wall-clock time from the start of the run to the completed output. A long runtime means the reconciliation stays active longer, costs more on a usage-priced machine, delays downstream settlement work, and raises the chance of hitting a job timeout. Runtime is the number people notice first, but it is not enough on its own.
RSS, or resident set size, is the peak amount of the process's memory that stayed in RAM. A high RSS can trigger swapping, an operating-system kill, or a container eviction when it approaches the machine's memory limit. A lower RSS leaves room for the operating system, other services, and temporary buffers. Partitioning reduces this peak, but it spends more time reading and writing temporary files.
Throughput is the amount of input processed per second. Here it is calculated as left rows + right rows divided by runtime. Higher throughput usually means the engine finishes a fixed workload sooner, but it can hide a resource trade-off. More workers raise throughput while also increasing RSS, CPU contention, and temporary disk traffic. A very high number from a run that discards output is also not a production guarantee. It has to be read with the output mode, hardware, schema width, and correctness result.

So the goal here is to answer one question: which run finishes within the required time while keeping memory, CPU, and temporary disk usage inside the deployment budget and producing the same correct result?

Architecture v1: fast lookup, expensive memory

For the first engine version, I optimized the inner match operation. The engine streamed the right file into a memory-resident hash index, streamed the left file, looked up each normalized reference, classified the candidate, and wrote the result. A hash lookup is a good fit for exact reference matching, so the design looked strong in early tests.

However, the execution model was not optimal. The right-side index retained rows, reference buckets, duplicate state, and the values needed by the output path. Wide records increased the cost of every one of those structures. So a fast lookup did not make the working set small.

The initial design stayed because it remained the simplest and fastest path for small and medium files. It also provided the reference behavior the parity tests needed. The trade-off was a memory curve tied to the size and shape of the right-side input.

I expected a memory-bound design, but not to hit its ceiling this soon. When the benchmark moved from narrow fixtures to realistic exports, the cost showed up fast.

At 5 million total records, the width fixture held the row count constant and varied only the schema width. The table below shows how runtime and peak RSS climbed as the fixture grew from 15 to 30 columns:

Schema width	Runtime	Peak RSS
15 columns	21.09 s	60.8 MB
20 columns	23.65 s	67.8 MB
30 columns	31.62 s	81.3 MB

The real-data test surprised me even more than the synthetic benchmark.

The width increase did not merely add parsing time. It increased the amount of data the execution model had to carry while matching. So the next design had to bound the working set before it added concurrency.

Architecture v2: partition the working set

Partitioning came to me while I was thinking about how the engine loads data. If the engine loads one side and compares it against the other, then I do not need the whole side in memory at once. I can load one part of a side, run the reconciliation against it, and move to the next part.

The first idea was to partition one side only. However, that left the other side as one large file, so the problem was only half solved. Partitioning one side leaves two bad choices: load the complete other side into memory, or scan the complete other file once for every partition. So the goal became simple: partition both sides, then optimize the matching, which is the easy part once the working set is bounded. This shape also turned out to be useful for multi-source reconciliation.

So instead of one scan, the engine makes two staging scans. First, it reads the left CSV and writes each row to a left partition file. Then it reads the right CSV and writes each row to a right partition file. At the end of staging, neither complete source is resident in memory.

The engine then processes one partition number at a time. For a reference pass, it opens left-003 and right-003, builds an in-memory index for the right partition, streams the left partition against it, writes a result chunk, and releases that local state. Grouped passes sort and merge the two partition files while keeping only the current group.

Staging both sides is what lets the engine pair partition 003 with partition 003 and scan each source exactly once.

The partition selector uses the configured matching or grouping column. The staging hash applies the same trimmed key rule on both sides, so a candidate for reference ORD-2026-84921 routes to the same partition in each source. This routing rule is a correctness requirement, not a claim that all candidates load together at once.

The partitioned backend stages both CSV inputs into temporary partition files. It then processes one matching partition pair at a time:

The parser normalizes the configured key.
A hash selects the partition.
The engine builds a local index or an external grouped sort for that partition.
The partition emits typed results and carry-forward metadata.
The temporary working set is released before the next partition.

For grouped reconciliation, external sort and merge-read keep only the current left and right groups in memory. The cost moves to temporary disk and extra passes over the data. That trade-off is deliberate. Disk is slower than memory, but it is easier to budget than an unbounded process-wide index.

Partitioning reduced peak memory, but serial partition processing left CPU capacity unused. It also introduced temporary files, cleanup paths, sort failures, queue design, and skew. So the new architecture solved the memory problem and created independent work units at the same time. Those independent units are what made parallelism possible, and this is where Go became useful.

Architecture v3: parallel partitions with backpressure

Parallelism before partitioning would have multiplied the memory-bound design. Each worker would need its own index, or every worker would contend for a global one. Either choice would make the original failure mode worse.

Partitioning changed the shape of the problem. Each partition became a bounded unit with explicit input files, a local index or grouped merge, a typed result chunk, and a summary. So the scheduler could assign independent partitions to workers while keeping the final writer single-owned.

For the parallelism, I started with a worker pattern where each worker reconciled and sent its results straight to the result writer. The idea sounds fine, but it broke quickly. When many workers produce results faster than a single writer can consume them, the writer becomes the bottleneck, and in my first version the writing was very slow. That is where the writer crashed under load.

Adding a plain queue does not solve the problem on its own. The queue is a backpressure mechanism, not a second result store. If the writer is slower than the workers, the bounded queue fills and the workers pause. Without that boundary, parallelism would recreate the memory problem by accumulating completed results in process memory.

Watching the writer stall against the disk sent me reading about how a printer handles the same problem. A printer cannot accept data as fast as a program produces it, so the operating system spools the work to disk and feeds it out in order. That design is called SPOOL, Simultaneous Peripheral Operations On-Line, and it is exactly the shape I needed. So the split idea came from there.

So workers do not send every result event through a channel. Using the spooling design, each worker writes its typed events to a private disk-backed chunk and sends only a small descriptor to a bounded queue. The descriptor holds the partition number, the chunk path, the summary, and the carry-forward metadata.

What parallelism accelerates, and what it does not

The parser remains a sequential stream for now. --partition-workers starts after staging has already read the source files, so adding workers does not make one CSV file parse in parallel. The release comparison shows only a small parse change, from 2.14 seconds in v0.3.0 to 1.85 seconds in v0.4.0 and 1.83 seconds in v0.4.1 for the same 500,000-row left file. The parallel feature is not a parser optimization.

The matching stage is different. On a fixed 1 million-record, 20-column fixture, the partitioned single-source run moved from 24.41 seconds with serial workers to 19.94 seconds with four workers. A three-source run moved from 23.85 seconds to 19.65 seconds. The measured speedups were 18.3 percent and 17.6 percent. The result does not support a claim that multi-source is always faster with parallelism. It supports a narrower conclusion: independent partitions improve the reconciliation stage, while parsing stays outside the parallel region.

The memory trade-off is visible too. Single-source peak RSS rose from 36.7 MiB to 108.0 MiB with four workers. Three-source peak RSS rose from 34.7 MiB to 57.6 MiB. The queue bounds completed descriptors, but each active worker still owns staging, indexes, sort buffers, or result chunks. So more workers trade elapsed time for memory and temporary I/O.

Comparing the releases

To compare the improvements on realistic data, I built a small project called Paybridge to generate files closer to the transactions I handled on past projects. These fixtures are not official provider exports, and they do not represent every customer schema. The measured records include both sides of each reconciliation, so 20 million total records means 10 million left rows and 10 million right rows. Grouped passes behave differently from exact reference matching. Filesystem speed, cache state, CPU count, schema width, key skew, and output format all affect the result.

I built the same v0.3.0, v0.4.0, and v0.4.1 binaries and ran each against the same 1 million-record, 20-column clean dataset. Each run used the memory backend, NDJSON output discarded at the process boundary, and the same Apple M1 Pro host.

Release	Parse 500k rows	Reconcile 1M records	Reconcile peak RSS
`v0.3.0`	2.14 s	5.74 s	726 MiB
`v0.4.0`	1.85 s	4.22 s	723 MiB
`v0.4.1`	1.83 s	4.15 s	705 MiB

The parser improves before parallelism exists because other parser and allocation changes landed earlier. Parse time falls 14.5 percent from v0.3.0 to v0.4.1. Reconciliation time falls 27.7 percent over the same comparison, while v0.4.1 is only 1.7 percent faster than v0.4.0 on this memory-backed single-pair fixture. Its main parallel feature is not active in this comparison. The larger architectural gain appears when the partitioned backend is enabled, where the working set is bounded and partitions become schedulable.

Performance means nothing without correctness

A fast reconciliation that returns the wrong answer is worse than a slow one, because someone downstream trusts it. So every architecture change had to clear the same bar before it was allowed to be faster: produce the same result as the simple memory design, on every fixture.

The engine checks this with a parity suite that compares more than summary counts. It compares row-level events, matched counterpart ownership, grouped members, anomaly classifications, duplicate annotations, and the final summary between the memory backend and the partitioned backend. A partition that reorders work is only correct if it lands on identical output.

What I am still working on

The engine is not finished, and the parser is the next target. The staging scan is sequential, and it can skew badly on uneven data. Three ideas remain on the list:

Bloom-filter pruning over left keys, so a right row with no possible match is dropped before it is ever staged.
Adaptive broadcast joins, so a small side that fits the memory budget skips partitioning and runs in a single pass.
Binary staging, so the engine stops parsing each CSV twice.

Each one attacks a different bottleneck, and they will probably become the next article.

Final thoughts

The most useful result of this project was not the final runtime. It was learning that the design that looked fastest on convenient data was not the design I wanted to run in production.

Go turned out to be a good fit for that path. Once partitioning had carved the work into independent units, goroutines, channels, and a bounded queue turned divide-and-conquer into a scheduling problem rather than a memory gamble.

The next steps are the three changes above, plus more benchmarks and correctness tests, since parallelism raises the risk of race conditions.

The engine is open source. You can read the code, the release history, and the benchmark setup in the Reconify repository. If it is useful to you, a star helps other teams working on reconciliation find it.

I am not a designer, but let me tell you something about product experience

Mangabo Kolawole — Wed, 08 Jul 2026 04:17:22 +0000

I am not a designer by profession. I am a software engineer with over six years of experience, and I have worked closely on consumer products in delivery logistics, fintech, and healthcare. Long before I touched Reconify, I understood the importance of user experience, even as a backend engineer. I wrote about why backend engineers should learn about user experience back in 2022, and that belief still holds.

During my career, I have had the privilege of working with talented designers like Emmanuel and Olafemi, and each of them has shaped how I think about building products. I also took part in UX workshops and read Refactoring UI.

Along the way, I understood one thing: product experience isn't about making interfaces simpler. It's about making someone's work easier. Design gives you the tools to do that.

Two stories taught me that lesson the hard way.

Engineering should never dictate experience

At my first job, we spent six months building a food delivery product called Quatro, similar to Uber Eats. The product depended on real-time notifications for riders, and we built it on Django WebSockets during the early days of asynchronous Python. We still shipped it for testing among consumers, riders, and restaurants.

The first problem was technical. The WebSocket notification system was buggy and did not alert riders quickly enough. Orders sat waiting. Riders didn't know when to move. Restaurants called support asking why no one had picked up the food.

The second problem was more interesting. Even after we fixed the notification bugs, the workflow still didn't feel natural for riders or restaurants.

We removed the WebSocket-based notification system that same week. Instead of making riders wait for a push notification before picking up an order, we replaced it with a single screen listing orders as new, ongoing, or cancelled, refreshed at a high rate. That screen ran for 12 months, and WebSockets only came back a year later, this time in read-only mode. By then, we had learned enough about the experience riders needed to design the engineering around it, instead of the other way around.

Restaurants, especially the busiest ones, were focused on cooking and serving customers. They didn't have the attention to spare for a tablet buzzing with new orders. We had designed the workflow around what we expected restaurants to do, not around what they actually had time to do. So we moved order-management responsibility to the rider, who was already coordinating the pickup.

The lesson: engineering should never dictate the product experience. We copied Uber Eats and Glovo's technical implementation, WebSockets and all, before validating that we had the same operational constraints they did. The question we should have started with wasn't which technology they used. It was what experience our riders needed, and what the simplest technology was that could deliver it reliably at the time. We never even showed the riders the application before they started delivering with it.

The second lesson: put yourself in your customer's shoes before you build a prototype. That step saves time, even after you hand the shoes back and watch how they actually walk in them.

No matter how much research you do, you'll still make assumptions. Some will be right, others won't.

The first version of your product isn't there to prove you were right. It's there to reveal where your assumptions break down.

That's why observing users is just as important as empathizing with them. Research helps you build a better first prototype, but observation is what turns that prototype into a great product.

I thought I'd learned that lesson. Six years later, while building Reconify, I found out I'd only learned half of it.

Know when to build for trust or for speed

Over the following six years, I worked on fintech, fiber network management, construction software for European companies, healthcare, and edtech products. Those experiences refined my understanding of user experience, but they never challenged its foundations.

I am currently working on a financial reconciliation software for fintechs, called Reconify.

The idea started with audit work I did for mobile money companies in West Africa, matching records between network operators, accounting systems, and payment processors. That work stayed with me because reconciliation touches everything a finance team does, and almost nobody enjoys doing it by hand. Reconify became the system finance teams use to reconcile those transactions at scale.

Using AI, I vibe-coded the first version of the Reconify dashboard in a few days. Before I trusted it, I put myself in a finance analyst's shoes: watching how analysts reconcile transactions in Excel, testing the competing tools I could get access to (most hide behind a demo request instead of a free trial), and talking with people at fintech companies like Djamo and Conduit about where it hurts most. What I learned mattered more than how I learned it.

That research taught me one thing: reconciling 100 transactions is not that different from reconciling 40 million, because it is a scale problem, not a logic problem. Matching turned out to be the least difficult part of reconciliation. The harder work happens before it, during ingestion and data cleaning, and after it, during adjustments, investigation, and audit. That insight redirected where I put my energy.

With that research behind me, I made a handful of design decisions for the first version of Reconify. Two of them turned into the stories that taught me the most: how analysts wanted to review exceptions, and how they wanted to upload their data in the first place.

I believed I had translated the research into the right decisions. Testing showed me how far I still was from that.

I have a close relative who works as a finance analyst. I showed her the platform, helped her generate realistic files, and watched her use it.

She reached the upload step and wanted to add every source at once, not one file at a time. Sequential uploads felt slow to her and did not match how she works. I replaced the step-by-step modal with a drawer, large enough to hold several sources in one pass.

When she opened an exception, she expected a panel that let her keep browsing the table while she worked, not a pop-up that blocked it. That confirmed a decision I had made without testing it, the exception panel instead of a modal, and showed me why the choice mattered.

I thought I had already done the work, through research and design decisions made in good faith. Testing showed me that good faith is not the same as good judgment.

I had built for correctness and completeness rather than for trust.

Analysts do not want fewer clicks. They want certainty that the number in front of them is right. They will take an extra confirmation screen every time, which protects them from a reconciliation mistake.

Consumer products optimize for speed. Professional tools optimize for confidence.

If you are a software engineer or a founder, invest in learning UX

If I could start again, I would spend less time looking for clever technical solutions and more time understanding the people I was building for.

Today, building software has never been faster. That doesn't mean it's easier, just that it's easy to turn ideas into a working prototype nowadays, whether you have the skills or not. But one principle hasn't changed: software exists to solve problems. And understanding the problem hasn't become significantly faster.

As engineers or founders, we naturally fall in love with solutions. We discover a new technology, an elegant algorithm, or a clever architecture, and immediately start thinking about where we could use it. I still catch myself doing exactly that.

Instead of starting with a solution, start with the people.

Who is struggling?

What are they trying to accomplish?

What gets in their way?

Only after answering those questions do I think about the technology that could help.

And in a world where everyone can build software and create slop faster than ever, I believe the ability to observe, understand, and design around real problems will remain one of the most valuable skills an engineer or founder can develop.

If AI scares you, learn software architecture

Mangabo Kolawole — Sun, 19 Oct 2025 15:14:56 +0000

AI demos are getting scary good.🥶

I've seen developers use AI to migrate an infrastructure of over 200 lambdas to Flask code in days instead of months. That's not a minor productivity boost – that's the kind of work that used to justify entire teams.

If you're worried about your job, you're paying attention.

But here's the thing: AI can only replace developers when stakeholders know exactly what they want and users know exactly what they need. And we all know that's never happening. Still, some developers are already getting replaced.

It's not the best developers getting replaced. It's the ones who only know how to write code. The ones who can't make decisions. The ones who've never learned software architecture.

Can't AI just become a software architect too?

You'd think so, right?

It depends.🧐

And "it depends" is the most powerful phrase in software architecture. It's also the phrase AI struggles with most.

When I read "The Fundamentals of Software Architecture" by Mark Richards and Neal Ford, one section stuck with me: the eight core expectations of a software architect. I started checking them against what I've seen AI actually do:

What AI can handle:

Make architecture decisions – It can generate options and trade-offs, but it doesn't understand your business constraints or politics.
Analyze the architecture – Actually pretty good at spotting dependency issues and technical debt.
Keep current with trends – Trained on everything public (though its knowledge has a cutoff date).
Ensure compliance – Can check rules, but doesn't feel the weight when things break.

Where AI falls apart:

Business domain knowledge – Needs you to explain everything explicitly, can't pick up on unspoken context.
Interpersonal skills – Can't mentor a junior dev through a tough bug or read the room in a meeting.
Navigate politics – And this is the big one.

Here's what navigating politics actually looks like: You're in a room proposing microservices. The CTO wants them because they're trendy. Your team lead hates them because of past trauma. The product manager doesn't care but needs the project done in 3 months. You know microservices will take 6 months and cost 3x more to run, but a modular monolith could ship in 2 months (you'll say 3 to the PM anyway – things can go wrong).

Can you hold your ground? Can you explain the trade-offs without making anyone feel stupid? Can you find the compromise that gets everyone on board? Can you take responsibility when you're wrong?

AI can't do any of that. It doesn't have to live with its decisions. It doesn't have to look someone in the eye and say "I was wrong, here's how we fix it." It can't mentor the developer who's struggling to implement the architecture. It can't lead a team through a failed deployment at 2am.

Those skills require something AI doesn't have: stakes, consequences, and the ability to actually care about the outcome.

So yes, AI will be a limited software architect. A very useful tool, but a limited architect.

And if you're worried about your job as a software engineer, learning software architecture might be the best career move you can make right now.

So how do you actually become a software architect?

Start by reading. You need to know if you'll actually like this work before investing months into it. I started with "[The Fundamentals of Software Architecture](https://a.co/d/dl18Yen)" by Mark Richards and Neal Ford. It's not a promotion—it's just what worked for me. The book taught me two things: the technical patterns (microservices, event-driven, layered architectures) and what the job actually involves (spoiler: it's a lot of meetings and explaining the same decision five different ways).

If you finish it and think "yes, I want to do this," move to "Designing Data-Intensive Applications" by Martin Kleppmann. Then the "System Design Interview" books (both volumes). These aren't just interview prep, they're where you learn to think in systems, not just code.

Then get your hands dirty with certifications.

Here's why you need them: certifications force you to learn the full toolkit of a platform, not just the three services you use at work. And whether you like it or not, having AWS Solutions Architect plus database and orchestration certs proves to clients and employers that you can actually integrate these systems.

Pick one cloud provider and go deep. AWS, GCP, Azure – doesn't matter which, but commit to one. I used ExamPro for my AWS Solutions Architect cert. The certification itself matters less than the process of learning how these systems actually work at scale.

Get certified in databases. MySQL or PostgreSQL for relational, MongoDB for document stores. Then explore the big data side – Clickhouse, Cassandra, whatever's relevant to your domain.

Learn the orchestration stack: Docker, Kubernetes, Terraform. You need to understand how systems get deployed and managed, not just designed.

Don't skip security either. Read up on it, understand the fundamentals. You don't need a CISSP, but you need to know enough to not design systems that leak customer data.

But here's the part most people skip: practice with real constraints.

AWS gives you $1,000 in credits for startups. Use them. Don't just build a todo app, reproduce the problems big companies actually face.

Build a system that handles 10,000 writes per second. Watch it struggle. Then figure out how to fix it – sharding? Caching? Different database? Let it run under load and see where it breaks.

Or pick a real company problem: How does Twitter handle the timeline for users with millions of followers? How does PayPal process payments without double-charging? How does Netflix serve video to millions of users without going bankrupt on bandwidth costs?

Try to implement those solutions in AWS or your favorite cloud platform. You'll fail. That's the point. Then you'll understand why those companies made the architectural decisions they did.

This is how you learn what "it depends" actually means. A solution that works for 100 users dies at 10,000. A database that's fast for reads becomes a nightmare for writes. You can't learn this from books, you have to break things.

And use AI as your sparring partner.

I have a favorite prompt: I tell Claude to act as "a software architect with 20+ years of experience who thinks he knows everything, hates change, and is extremely difficult to convince."😅

Then I pitch my architecture decisions to him.

If I can convince a stubborn AI that my approach is sound, I have a better shot at convincing a real skeptical CTO. It forces me to think through every trade-off, every edge case, every "but what about..." question before I walk into a meeting.

Start with the book. Try the prompt. See if this is the work you want to do.

If you've got questions about software architecture or want to talk through a decision you're wrestling with, leave a comment below. I'm always up for a good "it depends" conversation.

If you enjoyed this article and want more insights like this, subscribe to my newsletter for weekly tips, tutorials, and stories delivered straight to your inbox!

I think you should let AI write your code

Mangabo Kolawole — Sun, 05 Oct 2025 13:46:32 +0000

I've been skeptical about AI coding since it started gaining traction in 2022. And then, I changed my mind: I went from manually writing most of my code to having AI generate 90% of it. And honestly, I like the velocity.

In 2023, I was generating approximately 40% of my code using ChatGPT. The experience was frustrating. I'd spend more time explaining my codebase to ChatGPT than actually coding. The LLM would give me generic solutions that barely fit my project's structure. I mainly copied and pasted boilerplate and tweaked values.

However, ChatGPT excelled at backend work. When I built my first Python package, drf-api-key, ChatGPT handled about 60% of the work. It figured out Fernet encryption, structured the code properly, and saved me hours of research.

The real breakthrough came during my 8-month sabbatical when I discovered Cursor. I was introduced to the tool, which helped me quickly create boilerplate startup backends and handle infrastructure work. Those first sessions felt like magic.

Cursor understood existing codebases in ways ChatGPT never could. It could read patterns, maintain consistency, and actually improve code instead of just generating it.

But even Cursor had limits. It struggled with newer frameworks and couldn't access updated documentation (this is now possible with Context7). I'd still write configurations manually because Cursor would generate completely wrong setups.

Then November 2024 happened. MCP servers launched, Anthropic improved their coding models, Claude Code arrived, and Gemini 2.5 is actually very efficient. Now 90% of my code comes from AIs, and I've found ways to use them beyond just writing functions.

If you're not fully leveraging AI for coding yet, here's why you should start.

Before we dive in: I write about AI-assisted development, building products, and software engineering. If you want more content like this, subscribe to my newsletter.

AI coding feels like magic, but you are the magician

After 12 months of using AI for coding and watching other developers struggle with it, I've realized something uncomfortable: your experience with AI coding is a mirror of how good you actually are as a software engineer.

And I'm not just talking about your coding skills. I'm talking about everything:

If you're not methodical, your instructions to the AI won't work. The AI will generate garbage because your requirements are garbage.
If you don't understand your own codebase, you can't guide the AI to maintain consistency. You'll get code that works in isolation but breaks everything else.
If you skip preparation and planning, the AI will build the wrong thing perfectly. You'll waste hours debugging solutions to problems you never properly defined.
If you're impatient and want instant results, you'll give up after the first failed prompt instead of iterating toward the right solution.
If you have weak code review habits, you'll ship AI-generated bugs because you assumed the AI got it right.

I started with "magic eyes", thinking AI should know what I wanted. That led to frustration. You'd prompt the LLM once, get mediocre results, then blame the tool.

The reality: AI agents are like junior developers who are extremely fast but need clear direction. You wouldn't tell a junior dev "make this feature work" and walk away. You'd explain the codebase, show them patterns, and give specific requirements.

The 80/20 rule applies perfectly here. Software engineering is generally 80% preparation, 20% coding. Most developers skip the preparation and jump straight to prompting. Then they wonder why the AI generates garbage.

My workflow flips this: I use AI to help with the 80% (planning), which makes the 20% (coding) trivial.

When I get a task, I don't jump straight to coding. I involve the AI in the preparation phase. I give it context about the problem, the codebase, and the constraints. Then I let it help me think through the approach by drafting a detailed plan.

I let the agent draft detailed PRDs (Product Requirements Documents) for me. I don't write the PRD myself – I provide the agent with context, and it creates the plan. This forces me to think through what I actually need, and the agent structures it in a way that makes the implementation obvious.

My workflow:

Give the agent your ticket description, Figma designs, and documentation.
Have it draft a PRD and implementation plan.
Review and refine the plan together.
Let the agent cook, and iterate on the work.

The preparation phase is where you're actually being the magician. You're teaching the AI what you need, how you think, and what good looks like. The coding becomes almost mechanical after that.

The tools that changed everything for me

My current AI coding stack:

Cursor

My primary coding environment. It excels at understanding large, existing codebases.

I use Claude Sonnet 4.5 for actual coding and Gemini 2.5 Pro for PRD drafting or debugging complex issues. Where Cursor struggles for some reason, such as large-scale refactoring across multiple files, it can change code but doesn't always consider the broader architectural implications.

MCP Servers

This is the game-changer for 2025. MCP (Model Context Protocol) connects LLMs to external tools and APIs, giving them real capabilities beyond text generation.

My MCP setup:

Figma MCP - The agent reads designs directly and understands component structure.
Context7 - Gives agents access to up-to-date documentation for any framework.
AWS CloudWatch, Sentry, Resend - For infrastructure monitoring and notifications.

Claude Code

It handles infrastructure and deployment work.

I keep a CLAUDE.md file in my project root that defines my infrastructure standards and deployment workflow. Here's what the top looks like:

# Deployment Workflow
- Create timestamped database backup in /root/.backups-db/
- Pull latest changes from git (main branch)  
- Rebuild and restart Docker containers
- Run Django migrations and collect static files
- Verify all containers are healthy and services respond

## Critical Rules
- Never expose secrets in code or commits
- Always backup before deployment
- Use placeholders in documentation

When I deploy, Claude Code reads this file and follows the workflow exactly. It backs up the database, deploys the necessary updates, runs health checks, and emails me a report.

I also have a cron job running health checks every 6 hours. The entire observability pipeline is managed by Claude code and markdown instructions.

Where AI excels (and where it struggles)

Backend: AI's sweet spot

LLMs handle backend development exceptionally well. APIs, database schemas, business logic: these are structured problems with clear patterns, which is precisely where AI excels.

The key is to give your AI agent complete context about your codebase. Document your patterns, coding standards, and architecture decisions. When the agent understands how you structure projects, it maintains consistency across features.

For integrations, I feed the API documentation to my coding agent. It reads the docs, understands the authentication flow, and implements the integration. If the vendor provides an MCP server (like Sentry does), the agent can research implementation patterns directly. Otherwise, Context7 gives agents access to documentation for any framework.

The caveats

Always review what the agent generates. It might create code that works but doesn't scale, or overengineer simple tasks when explicit constraints haven't been set.

Tell the agent what to do, but more importantly, what NOT to do. Otherwise, they tend to become overly complex and generate unnecessary solutions. Be specific about your constraints: "Use the existing authentication middleware, don't create a new one," or "Keep this under 50 lines".

Frontend: Great, but not that great yet

Frontend development with AI is complicated. I code frontend 5x faster now, but the risk of creating unmaintainable code is higher.

My approach

I start small. I have the agent build individual components, not entire features. Build a button component, understand how it works, then compose it into larger structures.

For complex features, I use Figma's MCP server to provide the agent with visual context. But don't expect it to understand complete design systems from a single screenshot. Break designs into components and implement them piece by piece.

Frontend with AI requires more iteration than the backend. You need to review the generated code carefully and be willing to refine your prompts as needed. If that sounds tedious, stick to building components manually.

Infra: Surprisingly powerful

This surprised me the most. AI agents excel at infrastructure tasks.

When I ask Claude Code or Cursor to create VPCs, configure security groups, or write CloudFormation templates, it gets the details right more often than I do manually. Infrastructure is complex, and even experienced engineers rarely get deployments right on the first try.

What I've actually built

An automated trading system (7 hours)

I've been studying market movements since 2018 and recently took courses on algorithmic trading. The real work in algo trading is developing your own strategy: understanding indicators, backtesting parameters, and finding what actually works in live markets. That takes months of focused study.

But I wasn't going to wait 6 months to start trading.

Here's my approach: build the infrastructure first (money management, execution, monitoring), use third-party signals as training wheels, then swap in my own algorithm once I've developed it.

The problem with building trading infrastructure manually: the sheer number of integrations. Telegram API, message parsing, MetaAPI execution, email monitoring, database tracking, and deployment automation. Each one could take days to implement correctly.

With AI, I built it all in 7 hours. Now, while the system runs with external signals, I'm studying technical indicators and developing my own strategy. When I'm ready, I just plug in my algorithm, and the entire infrastructure is already battle-tested.

I'm building the boring parts now so I can focus on the interesting part: the actual trading strategy.

What it does:

Reads Telegram messages from trading channels.
Parses signals using an OpenAI agent into a format I want.
Validates trades before execution.
Places trades automatically via MetaAPI.
Monitors positions and adjusts stop-loss/take-profit levels.
Sends health reports every 6 hours.

How AI helped

Cursor built the Django backend (PostgreSQL + Celery) in a few hours. I provided the architecture documents and requirements, and it generated the API, database models, and background tasks.

Claude Code handles deployments and monitoring. I have a cron job that runs health checks:

claude --dangerously-skip-permissions --print "Perform a health check for the trading application and send an email report. Read /root/MONITORING_EMAIL_PROMPT.md to understand the format and information required. Use the sender email onboarding@resend.dev and recipient myemail@gmail.com. Important: Approve all file read operations and email sending automatically."

Here's an example email I received:

Claude Code reads the monitoring instructions, checks system status, and emails me detailed reports. The entire observability pipeline runs without manual intervention.

I even built an MCP server hosted on Gram. I just needed my OpenAPI document with the endpoints I wanted to expose, and Gram generated the MCP server.

Now I can monitor or place trades directly from Claude Desktop.

What I had to fix

Letting Claude Code handle deployments created some interesting issues. I receive signals from Telegram using Telethon, which creates a session file. On each deployment, that file was being deleted, causing my cron jobs to fail when trying to read messages.

It turns out that Claude Code was cleaning the file to maintain a clean state. Lost about an hour debugging that one. My solution: move the session file outside the project directory and copy it in during container builds.

Other issues I hit:

Hallucinated functions: Cursor invented non-existing validation functions. I had to force it to read the actual docs from Context7 and fix it.
Wrong environment config: Claude set USE_SQLITE=true in production, so my data was going to a file instead of PostgreSQL. Thankfully, I run backups before every deploy and restored the data.

MCP configuration generator (45 minutes)

I built a lot of MCP servers and got tired of manually writing configuration files for different tools.

I gave Cursor the MCP specification and examples of configs I'd written. It generated a tool that outputs valid configurations for any MCP server I need. 45 minutes of work that saves me 15-20 minutes every time I start a new project.

The rules I never break

I use AI for everything: building POCs for technical writing, developing backends and frontends, and deploying infrastructure. Here are the two rules I always follow:

Rule 1: Understand the project, codebase, and requirements deeply

You can't give good directions if you don't know where you're going. AI agents amplify your understanding; they don't replace it. If you don't understand your architecture, your coding agent will generate code that works today but breaks tomorrow.

Rule 2: Never give AI agents destructive permissions

I configure IAM policies so my coding agents can create and update resources, but never delete them. When something goes wrong (and it will), the agent will try several solutions first. But after multiple failed attempts, it defaults to the nuclear option: delete and recreate. That's how you lose data.

I make one exception: sudo access on servers for infrastructure automation. It's risky but necessary for deployment scripts. Never in production, though, where I create specific user roles and define permissions.

Final thoughts

Writing code has never been my favorite part of software engineering. Building solutions to problems is what I care about. AI lets me spend more time on that and less time on boilerplate.

I spent 7 hours building a trading system that would've taken me weeks manually. Now I spend my time refining the strategy, not debugging Django models. That's what AI coding actually gives you: less busywork, more focus on the stuff that matters.

Pick something you've been putting off because it feels like too much setup work. Let AI generate the structure. Then make it yours. You'll figure out pretty quickly where AI excels and where you need to step in. The tools are here. The question is whether you're willing to change how you work.

And if you're already using AI to write a significant portion of your code, please share your experience in the comments: your approaches, tools, and what you've learned.

If you enjoyed this article and want more insights like this, subscribe to my newsletter for weekly tips, tutorials, and stories delivered straight to your inbox!

Resources

Tools mentioned in this article:

Cursor – AI-powered code editor.
Claude Code – Infrastructure automation with Claude.
Context7 – Documentation access for AI agents.
MCP Protocol – Model Context Protocol specification.
Gram – MCP server hosting.

Further reading:

Claude Code Is All You Need by Gareth Dwyer – Deep dive on infrastructure automation.
MCP Quickstart Guide – Getting started with MCP servers.
Anthropic Prompt Engineering Guide – Writing better prompts.

Why I still use Django for my Saas Projects in 2025

Mangabo Kolawole — Sun, 18 May 2025 16:28:08 +0000

Today, it's easy to believe Django is outdated. Trendy tools and stacks like Next.js, Supabase, Astro, and T3 Stack are everywhere. They look fast and modern and come with many out-of-the-box features. You can spin up a full-stack app in a few hours and get moving quickly, doing the most important thing as a builder: providing value.

However, when I need to build a real SaaS backend that requires control, structure, and long-term maintainability, I still opt for Django as my first choice.

It's not always trendy, but it's dependable. And paired with the right tools, it's incredibly powerful.

What the Modern Tools Get Right

Let's be honest, Next.js and similar tools are impressive. You get frontend and backend in one place, deployment to Vercel is just a click away, and libraries like Prisma or Auth.js make setup fast and easy.

If you're building:

A landing page
A prototype
A low complexity SaaS MVP

...the JavaScript stack works great. I've used it myself, and it helped me ship quickly. But I don't like the chaos that emerges once your application grows. And that's what I think Django manages better.

What Django Still Does Better

If Django and other tools are featured in "The Tale of the Rabbit vs. the Turtle," I believe Django is the turtle. When your product starts needing:

More complex models
A real API surface
Fine-grained permissions
Long-term control over your backend

...the fast-moving frontend-first stack starts to feel limiting.

Django, on the other hand, forces you to be explicit. You define your models, serializers, views, and routes. It may seem like more work upfront, but that structure pays off. Your logic stays clean. Your code is easier to debug. And when someone joins the project later, they don’t have to guess where things live.

Plus, with AI tools like Copilot, GPT, and Cursor, you can generate most of the “boilerplate” code in seconds:

“Write a Django REST Framework ViewSet for this model and register it in the router.”

What used to be repetitive is now just a prompt away. Finally, the turtle is more like a ninja turtle at the end of the day.

However, before stating that, when building my first API SaaS with Django, I encountered a problem: distributing API keys to our users.

API Keys with Django Is Hell

And no, JWT can't be used for API Keys. This is my opinion. One issue I see often in SaaS APIs is how developers handle authentication. Many organizations default to JWT tokens for all applications, users, machines, internal services, and partners.

But JWTs were made for user authentication, not for public-facing API access. But said like that is a little bit abstract, but here are some reasons why I don't use JWT for API Keys:

1. JWTs are Meant to Be Ephemeral

JWTs are typically issued after a successful authentication (login), and they:

Contain encoded claims (like user ID, scopes, etc.)
Have an expiry time (often 15 minutes to 1 hour)
They are meant to be refreshed often via a refresh token

And there’s a reason for that: you can't revoke a JWT once it’s issued unless you maintain a token blacklist or rotate secrets aggressively. Its self-contained nature makes validation fast, but it also means that once a token is issued, it remains valid until it expires, regardless of the circumstances.

That's why short lifetimes and regular rotation are essential. But that’s also precisely what makes JWTs a poor fit for API key use cases, where keys might be issued once and used for weeks or months by machines or third parties. Adding a revocation system means registering the JWTs in the database, which is an anti-pattern to how JWTs are supposed to work.

2. JWTs Encourage Over-encoding

I've seen a codebase where developers often packed too much info into a JWT when using them as API keys:

{
  "sub": "org_abc",
  "role": "super",
  "scopes": ["read", "write"],
  "feature_flags": ["beta_search"],
  "tenant_id": "xyz"
}

When the API Key is no longer used for permissions and authorization, it becomes a moving target with encoded business logic. Any mistake in token generation can compromise authorization and pose a security risk.

API keys should be:

Opaque
Simple
Tied to metadata on the application, not logic in the client. The application handles all the metadata, what the key can do, who owns it, and whether it's active. The key itself has no logic or meaning embedded in it.

3. No Rotation Strategies

Well, imagine you've upgraded your authentication and API authorization system. With JWTs, you change the signing secret or key pair, and that's it. Great right?

Now, your clients are unhappy because their integration has stopped working suddenly. Every JWT you've issued prior has become invalid. With JWTs, there's no built-in way to support multiple active keys. You either:

Accept downtime and reissue tokens, or
Build a custom solution to manage various keys and key IDs (usually with extra middleware or a key registry)

That adds extra complexity to rotating credentials. Good API key systems should:

Use key IDs to support multiple active secrets
Allow graceful key rotation without breaking every client
Can issue keys with fine-grained scopes or rate limits

You can't do this easily with standard JSON Web Tokens (JWTs).

It's essential to note that JWT can be easily customized to address these issues, as you will need to implement or correct these flaws manually. But for me, it's altering the standard. JWTs work great for users.

Coming back to Django and API keys, when I was building my first SaaS, a payment aggregator, I needed to implement API key authorization into the API. Thank God there was this package: Django REST Framework API Key. It was actually great, easy to implement, and got things working quickly.

However, I ran into a performance issue. The package generates a key representing the API key and then stores a hash of that key. The hashing is performed using Django’s password hashing framework, which is designed for security rather than speed. It’s resource-intensive and slow.

There was no problem during key creation, but the issue showed up during permission checks. When the API key is passed in the header, the package hashes it and then looks up the key in the database. The process becomes relatively slow if you stack several permissions with different logic.

My first optimization was to change the password hashing algorithm to Argon2. That helped—request times dropped from five seconds to around 2.5 seconds. Still, that wasn’t good enough for me.

Additionally, the package didn’t support rotation, so logging and tracking had to be implemented manually. It also didn’t fully follow Django’s standard authorization pattern.

In Django, permissions are usually checked against the request.user, which is set during authentication. That gives you a lot of flexibility, even before permission logic is triggered.

And that’s when I decided to build something that I felt would be better and faster.

Building drf-simple-api-key

Fernet is a part of Python’s cryptography package. It provides symmetric encryption with built-in support for multi-key rotation, which means you can encrypt something with one key and later decrypt it with another, without breaking anything. That’s exactly what you need when rotating API keys without downtime.

So I integrated it into a package I built: drf-simple-api-key.

I started small, trying to improve performance and match Django’s permission system more naturally. But over time, I added support for:

API key authentication
Key rotation
Usage tracking and analytics

Here’s the core idea behind Fernet key rotation:

from cryptography.fernet import MultiFernet, Fernet

# Your current and previous keys
current_key = Fernet(b'CURRENT_SECRET_KEY')
previous_key = Fernet(b'OLD_SECRET_KEY')

# Use MultiFernet for seamless decryption
f = MultiFernet([current_key, previous_key])

# Encrypt new API keys using the current key
encrypted = f.encrypt(b"api-key:my-user-123")

# Decrypt any key (old or new)
decrypted = f.decrypt(encrypted)

If you rotate secrets properly, old keys continue to work, and new keys utilize the updated key. This avoids breaking every client at once, precisely the problem I faced.

To make rotation operational, I added a setting in drf-simple-api-key that signals Django to start a rotation window. When this setting is enabled:

The system generates new keys using the new secret
But it still accepts and decrypts requests signed with old keys
Once the rotation window ends, you can safely remove the old secret

This means rotation is smooth, and users don’t see any interruption: no mass invalidations, panicked clients, or backward-incompatible behavior.

It’s one of the most valuable tools I’ve built, not something I wrote entirely from scratch, as the Django REST Framework API key heavily inspires it.

The results? Went from 2.5 seconds to 100ms or under for requests. 🔥

However, let's take a moment to revisit Django's greatness.

Final Thoughts on Django and Its Architecture

One thing I appreciate about Django is how easily it allows for clean behavior extension through Python’s class system. You are not forced to write custom decorators or awkward middleware chains. You can rely on structured inheritance, define base classes, override methods, and call super() exactly where it makes sense.

For example, in drf-simple-api-key, I needed to integrate API key authentication with other features, such as usage tracking and rate limiting. Instead of rewriting the same checks in multiple places, I created a clean permission chain.

Here is a base permission that checks that an entity is active, which by default is the user.

class IsActiveEntity(BasePermission):
    """
    A base permission that only checks if the entity (by default, the Django user) is active.
    """

    message = "Entity is not active."

    def has_permission(self, request: HttpRequest, view: typing.Any) -> bool:
        return request.user.is_active

Now you want to create a permission that checks both that the entity is active and that they have a paid subscription.

You can extend the permission like this:

class HasActiveSubscription(IsActiveEntity):
    """
    Extends IsActiveEntity and also checks for a valid subscription.
    """

    message = "Entity is not active or does not have an active subscription."

    def has_permission(self, request: HttpRequest, view: typing.Any) -> bool:
        # First, run the base permission check
        if not super().has_permission(request, view):
            return False

        # Then add your additional logic
        user = request.user
        return getattr(user, "has_active_subscription", False)

And that's one thing I like about Django: you can write exactly what needs to be modified, make the call you want, and intercept the state when and where you need it. That level of control is incredibly valuable, especially when your business logic doesn’t fit inside a preset workflow.

This allows me to build logic in layers, using inheritance and composition, without having to rewrite core behavior.

This is a typical Django and Django REST Framework (DRF) pattern. You see it in views, serializers, permissions, and middleware. It is not unique to my package, but it is one of the reasons Django scales well when your project grows.

When I built drf-simple-api-key, this pattern helped me keep the logic modular. One class validated the key. Another tracked usage. Another applied limit. And all of it worked together through well-defined method overrides.

Final Thoughts

I still use recent and trending tools to build projects and sometimes SaaS for clients. I work with Next.js, the T3 stack, Laravel, Nest.js, Fastify, Hono, and much more.

But I think Django fits better. Maybe that’s just me.

It gives me more control. The stack is not fighting against me. And the support is solid. Django has real long-term support, LTS, a word I know the JavaScript ecosystem is unfamiliar with.

So, those other stacks are not automatically better. I think Django might be better than almost everything else out there. For me, it is one of the best software engineering tools ever made. That is not just an opinion; it is based on real experience.

That said, here’s the caveat.

If you don’t have the right tools around Django, don’t know the ecosystem, or are missing good packages, it can feel slow and heavy. Django does not hold your hand. It expects precision. Before you start, you need to know what you’re doing and what you are not doing.

And if Django doesn't fit your current project, that's fine. Use what works for you. Use FastAPI, Node.js, NestJS, Ruby, or any other framework that aligns with your goals. Too many excellent tools and libraries are available to become locked into a single mindset.

What matters is understanding what you're optimizing for: speed to market, maintainability, and the developer's skill set, and making your decisions accordingly.

This is what works fast for me.

If anything I said was helpful or you'd like to discuss authentication, API security, or Django packages, you can find me on Twitter. Feel free to reach out. And if you're trying to learn Django, I can point you to resources that helped me personally.

None of these are sponsored. Some are paid, some are free. I’ve bought them myself. They helped me understand things more clearly; I still reference them today.

Django documentation: Still one of the most well-written documentation of a tool, in par with Laravel.
Django course by Thinkster: Their course helped me understand DRF with concrete examples and also assisted in implementing JWT authentication. I'm unsure if the course has been updated, but the patterns are similar to what's done today.
Test-Driven Development with Django, Django REST Framework, and Docker by Testdriven: An ideal course if you want to understand how to build an API with Django, Docker, add testing, and CI/CD pipelines.
LearnDjango tutorials by Will Vincent: Vincent provides great tutorials and courses for learning Django. I followed the Django for APIs course, but didn't finish it because I was looking for a particular chapter. It's a well-written course, and you will find interesting resources on the website.

If you enjoyed this article and want more insights like this, subscribe to my newsletter for weekly tips, tutorials, and stories delivered straight to your inbox!

Is Software Engineering Still Worth It in 2025?

Mangabo Kolawole — Sun, 22 Dec 2024 14:00:00 +0000

Hi, I hope that you are doing well.

This will probably be my last article for 2024, and I hope you had a great year.

I had an interesting year, I would say—a lot of downs and bad, but thankfully, I think I am ending it on a high note, so I'm particularly happy about the achievements and what I accomplished this year. 🎉

So, I wanted to share what I think will make you a better developer in 2025. I plan to discuss many things, including AI.

You have people laid off every Friday. There are a lot of layoffs. Many companies are asking right now for people with better skills; developers are not wanted anymore (That’s a cap, by the way 🧢).

Is it worth learning software engineering in 2025? These are normal questions you should ask yourself, which is true. How do you ensure you can still live as a software engineer and not be replaced? 🤔

The Impact of AI on Software Engineering

First, let’s talk about the big boy in the room. Recently, your employer can finally employ AI for $500 monthly and replace you. They will finally be able to communicate with it quickly, ask it precisely what they need it to do, and get results without the usual back-and-forth.

AI won’t say, "We can’t do this; that will take time." I think this is a perfect tool for product managers.

It’s a slave. If someone is told, This is exactly what I want, execute it. I’m telling you guys, this is exactly what they wish to…

Anyway, let’s calm down with the jokes. But you understand that we talk a lot about being replaced by AI. If you had told me three years ago that we would be at the stage of OpenAI, ChatGPT, Devin, and many other tools right now, I would have said, No, you are tripping. Give me at least ten years.”

But the thing is, we are moving so much faster, to the point where many things are happening right now.

So, the first question is: will AI take over your job as a software engineer?

Suppose you are only a coder or a programmer, maybe if you are a beginner.

The thing is, software engineering is not only about coding. Coding represents about 30% to 40% of the work; the rest involves communication, planning, and ensuring the work benefits the company.

The easiest part of software engineering is actually writing the code. The harder part is understanding what to write, why to write it, and whether it’s necessary to write it at all. 💻

Becoming a Better Software Engineer

Let’s be honest, there’s a lot of things you can do in just one week, but it takes a lot of meetings, a lot of delays, because you have to think about things like: Does adding this feature to the application actually create value?

So those are a lot of questions you have to ask yourself as a software engineer. Can AI ask those questions? Can AI talk to the manager and explain why we should or shouldn’t add this feature and how it might affect the project or the users?

Will AI be able to come up with its ideas and put them out there to be helpful?

We already know that in terms of creativity, technically, AI cannot do that.

AI is quite capable of creativity, as my interpretation of creativity is copying. In my view, everything has already been invented. We are just building upon things. So technically, AI can be creative, but let’s not go there.

Preparing for the Future of Software Engineering

So my advice is: instead of being just a programmer, try to be a software engineer. At the end of the day, we are creating tools that will help people.

You must learn to review code, understand architecture, and contribute to high-level design decisions. 🏗️

You should also learn how to make money as a software engineer.

There are a lot of things you should learn to do to become a better software engineer. I understand that in some years, we might disappear—maybe I’m wrong here. But I don’t think AI can be a great software engineer right now.

So, if you are a good software engineer, you should not fear AI. Naturally, learn about the tool. Learn about AI. Don’t be afraid to jump into it, try out stuff, and be enthusiastic about the change. 🚀

Because I think the sooner you embrace it, the better you’ll understand it while you move to something else.

Many inventions have appeared in the world and replaced jobs. For example, people used to deliver newspapers, but with the rise of digital media, those roles became obsolete.

However, many of those individuals transitioned into new opportunities within the same industry, such as digital content creation, logistics, or other technology-driven roles.

They didn’t just disappear; they adapted, reskilled, and found ways to thrive in a changing world, often leveraging their existing knowledge in innovative ways.

Eventually, they had to reinvent themselves—and that’s okay.

After software engineering, I think something exciting awaits us. It will be much more interesting and complex than what we’re doing now.

So don’t be negative about it; just be ready for it.

Embrace it.

The next step is to focus on building impactful projects. Let’s discuss how to become better software engineers.

Final Thoughts

Next step is building. Let’s talk about how to be better as a software engineer.

Look, the market is savage right now.

Even if you are a general developer, if you do not come to the market as someone who can set up a complex application, scale it, and maintain it, you are not a good engineer.

I’m not telling you to lie or try to invent stuff, but the requirements to enter the domain are very high right now.

Before stepping in, you need to know many things and have done some complex stuff.

You indeed gain a lot of experience when you work in the field, but this is not exactly the case right now. They are waiting for you to have those notions already so you can come and apply them.

So how do you do that?

You have to create. And no, hell no. I’m not talking about to-do apps.

I’m not talking about weather apps (Bro, this is not 2019 anymore) or simple visualization apps. You have to do something complex.

When I talk about complex, I mean building something significant.

It might be a social media app with AI features, a compiler, or something equally interesting.

You can also try things like setting up a complicated infrastructure on AWS, simulating millions of requests to a server, and observing how it handles the load.

There’s no excuse not to learn about scaling, handling infrastructure, and understanding how these things work.

To see how it goes, you can reproduce existing architectures on a small scale.

There’s no time to wait to learn these things.

Take software engineering as a science. When approaching it this way, you realize you need theory and practice. 🧪

Sometimes, you practice on your own, sometimes on other projects, but you have to practice.

And you need to practice at a higher level than before.

Build something for yourself. Build something for your learning.

Do it, even if it’s a mini AWS or a mini data center set up in your apartment.

Try to build. Try to really build.

Now, on to making money as a software engineer.

This might be complicated but don’t stress about it.

It’s okay if you don’t know how to monetize your skills.

But if you learn to make money as a software engineer, that’s fantastic.💡

This is related to building things and putting them out there to sell.

Look, you might fail nine times out of ten, but one success can be very rewarding.

I make money by writing. I learn something, write about it, publish it, and people read it.

Sometimes, I get thousands of views; sometimes, I struggle to crack 100 views.

But at the end of the day, I feel happy about it. 😊

I also write for an agency, creating articles and content for companies.

And I’m thinking of launching my products.

So don’t stress about it— do something.

Finally, I want to say this: we must consider ourselves scientists in a rapidly evolving domain.

Be curious, embrace change, and keep learning.

Focus on hope and perseverance.

Whatever you are going through, know that hope can be the answer.

It’s ingrained in us as humans—it’s how we survive.

So keep learning, keep growing, and keep trying new things.

I wish you a Merry Christmas and a Happy New Year. 🎄

Let’s make 2025 a great year for growth as software engineers.

Take care and see you next year.

If you enjoyed this article and want more insights like this, subscribe to my newsletter for weekly tips, tutorials, and stories delivered straight to your inbox!

What's the Best Software Architecture?

Mangabo Kolawole — Sun, 01 Dec 2024 14:00:00 +0000

Hi there 👋

At the beginning of my career, my knowledge of software architecture was quite limited. To be honest, I had no idea about most complex software architectures. It was limited to MVC/MVT architecture from tools such as Django.

However, in my first year, I had the chance to work on a real-time oriented product, which was a food delivery service. That pushed me to learn more about complex architecture because it was quite challenging, and I had to work on many parts of the project.

Briefly, we were building a notification system based on Websockets in Python so riders, restaurants, and consumers could exchange information in real-time. Instead of working on software architecture only at a coding level, I was introduced to it on an infrastructure level. We had to set up real-time databases, ensure latency was as low as possible, and much more.

Over the years, this led me to experiment with new ideas to learn more about other software architectures. How are they used? Why are they used? What are the pros and cons? What is the best architecture?

hum

If you are interested in more content covering topics like this, subscribe to my newsletter for regular updates on software programming, architecture, technical writing, and tech-related insights.

Strange Things I Have Noticed

The first thing I noticed is that software architectures are mostly divided into two categories: synchronous architectures and asynchronous architectures. Depending on the needs of your project, you will have to choose an architecture that fits one of these categories. Actually, you can mix both.

The second thing I noticed is that there are many bad habits regarding software architecture. A lot of companies unnecessarily over-engineer or naively under-engineer.

Two years ago, I interviewed with a company that had an interesting project they had been trying to launch for two years. To be honest, I should have stopped the interview right there. However the project was intended to scale internationally at a certain point, and I loved the idea of leading this expansion. I hadn’t done this before, so it was definitely exciting. However, there was a big red flag. 🚩

Me: Tell me more about your architecture.

The CTO: Well, we have nine microservices right now...

Abeg

Oh, please! What do you mean you have nine microservices but no official launch? Having more microservices than users is crazy to me. That was a big red flag, and I said no.

I have had this experience quite a few times when I found myself discussing projects with startups that hadn’t written a single line of code yet but were already implementing complex software architecture without even an MVP. There is no need for that.

Now, I’m not saying over-engineering or under-engineering are inherently bad. They’re not. It depends on how and when you use them. And this brings us to the question we face even today:

What Is the Best Software Architecture?

The answer is short: It depends. It depends on your software goals, requirements, users, and business needs.

How to Choose an Architecture

Before discussing an architecture, you should discuss the problem and understand it. You should understand how the business and the users interact with the software. Most importantly, you should focus on the data.

The data. The data. And the data.

By understanding how data is created, read, and updated, you can gain deep insights into what you are trying to solve by building the software. This understanding will allow you to choose or craft the correct architecture for your system. 💡

There are many resources to get started with software architecture, such as online courses, articles, and books, but my favorite has been Fundamentals of Software Architecture by Mark Richards and Neal Ford. (This is not a promotion, by the way!)

It’s a book that introduces the basics of software architecture, presents different architectures, and discusses them in depth. Software architecture is a vast domain, but the book does a great job of getting you halfway there if you’re looking to learn more. The rest of the journey involves diving deeply into specific architectures, implementing them, and scaling them. 📚

I’m currently writing an article that serves as an introduction to software architecture—a 4,000-word guide to understanding the basics and exploring the most commonly used architectures. This guide will also include links to excellent resources to continue learning.

If you enjoyed this article and want more insights like this, subscribe to my newsletter for weekly tips, tutorials, and stories delivered straight to your inbox!

How Software Engineers Make Their Software Run Faster

Mangabo Kolawole — Sun, 10 Nov 2024 13:00:00 +0000

Last week, I was assigned a task to speed up an insertion script. The program was a straightforward DAG, running daily tasks to retrieve CSV files and process them into a database.

These files contain information about financial transactions, which can quickly become heavy due to the sheer number of rows—often reaching millions.

The script for inserting rows into the database followed a simple flow:

Read a row.
Perform checks to ensure essential data is present.
Attempt an insert.
Log errors for issues like duplication or invalid data.

Pretty simple, right? However, behind this simplicity was hiding a major issue.

The task was slow as HELL. 🤬

Processing a file with 4,000 records could take up to 20 minutes, which was far very very very from ideal.

This was where the hunt for the culprit began.🤺

The Suspected Causes 🔍

Latency? No, we had other tasks running on the database, and it wasn’t experiencing unusual delays. The database server was performing well, according to usage stats.

Disk usage? Yes, the machine's disk was under heavy usage due to large CSV files loading into memory. One solution could be processing the file in chunks rather than loading the entire file at once, which would help manage disk usage. However, this wouldn’t solve the core problem of the script’s speed.

So, what was wrong?

The Real Problem ⚡️

With my current knowledge, I knew that 4,000 individual inserts in the database would likely cause a bottleneck. We were looking at around 1.78 transactions per second.

After some research, I discovered that batching is a more efficient approach for inserting large numbers of rows. But why wasn’t it already in use?

Well, batching introduces a risk: if there’s an error in one record within a batch, it can invalidate the entire batch.

As it turns out, the developer that I know pretty well…

…who originally wrote this script opted to perform individual inserts to avoid this risk.

The problem with individual inserts is that they slow down significantly with large datasets because each insert requires:

Separate transactions
Frequent disk I/O
Repeated index updates

The Solution 🛠

To solve this, I decided to use batch inserts with a more refined error-handling approach. I was aware of the main types of errors that might occur, such as:

Missing consumer information
Duplicate entries
Invalid data

Duplicates were errors I could afford to ignore.

For missing consumer information and invalid data, I implemented a caching system. By retrieving all consumer IDs and storing them in a hash table cache, I could check each record's consumer information before insertion, removing an unnecessary transaction check.

For invalid data, I also included format validation to ensure only clean data was passed to the database, reducing the load of handling these errors.

To make batch inserts resilient to errors, I used INSERT ... ON CONFLICT DO NOTHING ... RETURNING ref. This way, the batch insert wouldn’t fail completely upon encountering an error, and I would still get the references of successfully inserted rows.

The Results 🚀

With these changes, I managed to reduce the processing time from 20 minutes for 4,000 records to an incredible 2-3 seconds. 🤯

Certainly! Here’s a refined version with minimal emojis, italics, and bold elements, staying true to your content:

We start by analyzing the problem thoroughly, identifying likely edge cases to understand performance challenges, and tailoring optimization strategies accordingly.

In my case, understanding that data from consumers could go missing and that there might be some invalid columns for certain rows meant I could perform checks without heavily involving the database.

So, *next time, do your best to understand a problem correctly, and optimizing it will come naturally.✌️*

Do you have a story to share about an optimization you made and you were happy about? Let us know in the comments👇

If you’re into more content like this, subscribe to my newsletter for regular updates on software programming, architecture, technical writing, and other tech goodies.

How to Quickly Navigate a New Codebase

Mangabo Kolawole — Sun, 03 Nov 2024 12:00:00 +0000

So you've just accepted the software engineering or developer job offer?

Well, first of all, congratulations on this big milestone! 🎉

Now your first challenge in the first weeks of the job is understanding the codebase. How do you know the work that’s been done so you can start contributing ASAP? What mistakes should you avoid? What are the right questions to ask?

I’ve worked in four companies, and I’ve also consulted for others as a developer or lead developer. Through these experiences, I started noticing patterns—patterns that helped me get better at understanding codebases and contributing from day one.

But hey, most of the time, life gives you your toughest battle on day one when you spend hours just trying to set up the project. Why? Because some tools aren’t compatible with your machine or some obscure configurations are nowhere in the documentation… 😅

But let’s stay optimistic and assume the best-case scenario—let’s dive into how you can start contributing ASAP.

In a nutshell, there’s one big piece of advice: understand the business cases. Let’s dive into it.

If you’re into more content like this, subscribe to my newsletter for regular updates on software programming, architecture, technical writing, and other tech goodies.

How Do I Contribute to Codebases on Day One?

I like to come prepared, so I usually know a good bit about the project before my first day. I already have an idea of the project’s state, and if it’s a public product, I’ve likely spotted some bugs and even thought of solutions. This prep helps me navigate issues and tickets, giving me a head start on figuring out where I can contribute.

And, believe it or not, setup issues can be a blessing! If I run into problems, I ask myself: Was this a skill issue or a documentation issue? Either way, I make a point of documenting it. If I had set up struggles, chances are someone else would too.

I also make it a habit to join architectural discussions. I’ve been lucky to work with companies that encourage open communication, where discussions are public and anyone can chime in with ideas or advice. This gives me an understanding of the bigger picture and lets me make meaningful contributions right away.

But before diving into all that, here is the rule I always follow.

Understand the Business Cases

As a developer or software engineer, remind yourself why you’re here: you’re writing software to solve a problem. In any company, these problems relate to customers or internal needs, and you’ll be implementing the business logic of the solution through design, architecture, and code.

The rule of thumb is: Focus on the what and the why.

The Company Has Great Documentation: Docs, Tests, and Comments on Code

If the company has a strong culture of documentation, you’re in luck. Whether it’s product documentation, architectural guides, or code comments, this will help you understand the business from a software perspective.

What does this look like IRL?

Imagine your first day at a fintech company.

Fintech companies are known to use domain-driven architectures. In a domain-driven setup, business cases and their software solutions are organized into domains, making it easy to work independently in one area without breaking down others.

Say you’re assigned to a team managing customer wallets. To understand this part of the code, start by asking: What’s the business goal here? What are the user stories?

In short, get to know WHAT this part of the software does and WHY it exists—without diving straight into the code, which is the HOW.

The Company Lacks Strong Docs but Has Tests

Not every company has great documentation. Sometimes, especially in fast-moving startups, documentation takes a backseat. But if there are good tests, you’ve got another way in.

Well-written tests explain what different parts of the software do. They’re like mini-documentation embedded in code. Find the tests and read them—they’ll help you understand the business cases and the codebase.

No Docs, No Tests, and the Only Dev Who Knows the Codebase is Leaving

The nightmare scenario. I used to think this was just a joke developers told to scare each other, but I’ve experienced it twice. For me, it’s a red flag, but if you’re up for the challenge, here’s what you can do:

Take the codebase as it is. Test the functions, run the app in a debugger, and even crash the code intentionally to see how it behaves.

If you don’t understand a portion of the code, try commenting it out or removing it temporarily to observe how the app reacts. If you’re familiar with a debugger, it’s your best friend for understanding how things work behind the scenes.

No matter what situation you’re in, follow these core rules to keep making progress:

Ask Questions — Lots of Them

Your first few days are the golden period for asking questions. You’re new, and everyone expects you to be a little lost. Use this time to dig into anything unclear and get the lay of the land.

The more questions you ask now, the fewer headaches you’ll have later.

If you don’t understand a piece of code, reach out to a peer or a senior dev. Don’t sit there feeling stuck when a quick answer could move you forward. Ask about the codebase organization, common issues, or any hidden quirks in workflows.

Notice any gaps or oddities in the documentation? Point them out. The team will appreciate your initiative in understanding the code.

Read, Read, and Then Read Some More

In the early days, the codebase itself is your best friend. Spend time going through it line by line to understand each module and its interactions. Here’s how to tackle it:

Documentation: If it exists, soak it up. Pay attention to architecture diagrams, core workflows, and explanations of common patterns.
Tests: Good tests are like treasure maps—they tell you what parts of the application are supposed to do. Run them, step through them in the debugger, and let them guide you.
Issues or Backlogs: Check open or recently closed tickets. They reveal what the team has been working on and the current pain points. You’ll get a sense of ongoing improvements and challenges.

Don’t Be Shy About Making Small Changes

Once you start getting a feel for the code, go ahead and make small contributions. These don’t have to be groundbreaking—small tweaks are a great way to build confidence. Look for low-hanging fruit:

Fix typos or add missing documentation: Small but helpful ways to make your mark.
Improve code comments: Sometimes code is written in a rush, and a bit of clarity helps everyone.
Tackle minor bugs: If you spot an easy fix, mention it to your manager or tech lead. You’ll be learning and making things better.

Be Patient with Yourself

Remember, no one expects you to know it all right away. Don’t pressure yourself to master the codebase in the first week. Keep learning, keep contributing, and stay curious.

Over time, you’ll find yourself moving from observer to contributor—and that’s when things start to get fun. 🎉

Conclusion

In short, getting up to speed with a new codebase is a journey. Ask questions, read everything you can, and make small, meaningful contributions. Take your time to understand not just the “how” but the “why” behind the code. With patience and curiosity, you’ll be making an impact before you know it.

Now, go make that codebase yours!

If you enjoyed this article and want more insights like this, subscribe to my newsletter for weekly tips, tutorials, and stories delivered straight to your inbox!

How Senior Software Engineers Document Their Project

Mangabo Kolawole — Sun, 27 Oct 2024 12:49:35 +0000

There’s one task that software engineers hate, yet this small attention to detail is what separates a good software engineer from a bad one: How do they document their project?📝

A few years ago, I was responsible for setting up a fintech project. Because we decided to move quickly, planning for scalability wasn’t a priority. Our focus was on validating the idea, so we pushed forward, creating APIs, architectures, and systems with simple solutions, not overly concerned about the future.

However, as the person in charge of the backend and infrastructure, I knew that while my memory was reliable, it wouldn’t be enough to recall all the details six months down the line.

In my research, I discovered a convention I liked: ADR, or Architectural Decision Record.

It’s essentially a document that traces all changes made to an architecture: the change itself, its impact, and what we learned from it.

Think of it as a personal journal but for the team.

Why is it important?

Humans forget: Documenting changes helps us because we easily forget the reasons behind choosing one architecture over another.
It makes the team better: Let’s say you’ve tried various solutions for an issue and documented the successes and failures. You learn from it, and others can too, even developers who come in after you.
Future developers will thank you: Imagine a dev coming to a codebase and trying to understand why a change was made five years ago. Somewhere, a developer is likely struggling with this because the previous engineer left without documenting it, and they’re not thrilled. Meanwhile, in another company, a developer finds an ADR explaining those changes, and they’re incredibly grateful.

How do you write one then?

There are several conventions to follow, but you can always adapt them to what works best for you.

The convention that inspired me is here: https://adr.github.io/madr/. You can also check Amazon’s ADR process here: https://docs.aws.amazon.com/prescriptive-guidance/latest/architectural-decision-records/adr-process.html.

Here is an example of a template you can use.

# Example Title: Database Choice for User Data

## Context and Problem Statement

We need a scalable database to store and manage user data efficiently as our user base grows.

## Decision Drivers

* Scalability
* Data consistency
* Ease of integration with existing services

## Considered Options

* PostgreSQL
* MongoDB
* Amazon DynamoDB

## Decision Outcome

Chosen option: **PostgreSQL** because it provides strong data consistency and aligns well with our need for complex queries.

### Consequences

* **Good:** Supports ACID compliance, enhancing data reliability.
* **Bad:** May require more tuning to achieve high performance with large datasets.

### Confirmation

We’ll confirm this decision through periodic load tests and performance reviews as the user base scales.

## Pros and Cons of the Options

### PostgreSQL

* **Good:** ACID compliance, robust community support.
* **Neutral:** Setup and tuning can be time-consuming.
* **Bad:** Lacks native horizontal scaling.

### MongoDB

* **Good:** Schema flexibility, horizontal scaling.
* **Bad:** No ACID compliance across collections, limiting data integrity.

## More Information

For additional details, see the database performance evaluation [here](link-to-evaluation).

This kind of document can be present within the project repository, or a notion, or JIRA.

In my last company, where I worked as a frontend engineer, we didn’t have a single document for all architectural changes.

Using GitLab issues and linking every change to an issue branch helped us track the reasons behind changes, even months after implementation.

This practice saved us countless times. As I always say, no matter how smart you or your teammates are—your CTO, manager, or anyone involved in the project—they won’t remember every technical decision made two years ago.

Unless, of course, you’re working with 10x engineers. 😆

Conclusion

And that’s it for this article. We have discussed how companies and tech team leaders use ADRs to document architectural decisions on their projects and how much it helps them, teammates, or even the people who worked there after they left.

If you have experiences to share or any thoughts on the article, feel free to drop them in the comments below.

I’m always open to feedback and happy to engage in discussions that can help us all learn and grow.

If you enjoyed this article and want more insights like this, subscribe to my newsletter for weekly tips, tutorials, and stories delivered straight to your inbox!

How to Choose the Ideal Database for Your App: Prototypes, App at Scale, and Event-Driven App

Mangabo Kolawole — Sun, 13 Oct 2024 13:35:03 +0000

Storing data is something humans have been doing for thousands of years.

From carving information on rocks to developing language, writing on paper, and eventually storing data on disks, it’s an essential part of how we preserve and share knowledge.

But enough of the history lesson—if you're a developer, whether beginner, intermediate, or expert, you've likely encountered SQL and NoSQL databases, even if you didn't realize it at the time.

So, how do you make a wise choice when it’s time to develop an application? Are you starting a new project from scratch? Is it an integration into an existing system? Are you working on a project that needs structured data and integrity? Or maybe you're developing an event-driven application?

There are plenty of use cases to consider when choosing the right database for your application. For the sake of simplicity, we’ll focus on three common types of applications and the best database options for each:

Small-scale or prototype applications
Applications needing data integrity and robustness
Event-driven applications

Small-Scale or Prototype Applications

If you're building an MVP or testing an idea, you'll likely understand the importance of making a quick prototype to validate your concept before committing fully. Speed is crucial at this stage, and the same applies to how you handle data storage. Here are two common challenges:

You need a simple database with a minimal setup that can be easily moved around without complex configuration—after all, you're just trying to validate an idea.
You’re unsure about the structure of your data, and you want flexibility for evolving fields without worrying about strict data types or rigid schemas.

Personal Recommendations

For the first case, I recommend SQLite. It’s a simple, file-based storage solution that doesn’t require setting up a server. All data is stored in a file, making it perfect for lightweight applications that need quick deployment.

You’ll often find SQLite in embedded systems and mobile apps because it’s fast, easy to use, and doesn’t require much configuration. Despite its simplicity, SQLite can handle a surprising amount of data, making it a solid choice for early-stage projects.

In the second case, MongoDB is an excellent choice, especially if you’re unsure about how your data will evolve. MongoDB is schema-less, allowing you to store and modify data without worrying about rigid structures upfront.

It’s easy to set up and works well for both small-scale prototypes and larger applications. That said, MongoDB has its trade-offs—at scale, you may run into performance challenges, and its lack of strict data constraints can lead to inconsistent or unstructured data. However, MongoDB’s flexibility is often worth the trade-off for a prototype.

In addition to SQLite and MongoDB, you might also want to explore other powerful tools for rapid prototyping, such as LevelDB, Couchbase Lite, Firebase Realtime Database, and Supabase. These databases offer unique strengths like fast key-value storage, offline-first capabilities, real-time data syncing, and easy-to-use SQL features, making them excellent choices depending on your project’s specific needs.

During the prototyping phase, the focus should be on speed and flexibility rather than strict data integrity or performance optimization. Once your prototype is validated, you can revisit your database decisions as you move into more structured and large-scale development phases.

This brings us to the next step: building applications that require data integrity and scalability.

Applications Needing Data Integrity and Robustness

As your project moves beyond the initial stages and becomes a more critical system, the need for data consistency, integrity, and scalability becomes essential. Whether you’re managing sensitive information or developing a business application with complex data relationships, your database choice now needs to support strict data management practices. At this point, two common scenarios come into play:

You have a clear understanding of your data structure and require a database that ensures data integrity, consistency, and complex querying capabilities.
Your application is growing in complexity, and while you still need flexibility, you also need the ability to scale while handling large amounts of unstructured or semi-structured data.

Personal Recommendations

For the first case, PostgreSQL is an excellent choice. It’s a robust, open-source SQL database known for its ACID compliance, meaning it guarantees reliable transaction processing. PostgreSQL also excels in handling structured data, supports complex queries, and offers full-text search capabilities.

It is ideal for applications that require strict data integrity and need to process large volumes of data reliably. Additionally, its support for JSON allows flexibility with semi-structured data, making PostgreSQL a versatile tool for growing projects.

In the second case, MongoDB continues to be a strong contender, particularly for applications handling big data or evolving data models. While often used in early-stage projects for its schema-less design, MongoDB can scale horizontally to manage vast datasets across multiple servers.

It is widely used by large-scale companies like Uber and Toyota, which rely on its ability to handle unstructured data and adapt to changing requirements. That said, as your application grows, you may need to carefully manage MongoDB’s performance and data consistency, especially if you’re dealing with highly critical data.

While PostgreSQL and MongoDB are popular choices, there are other SQL and NoSQL databases worth considering for applications needing robust data integrity and scalability. MariaDB and Oracle SQL are both highly regarded for their performance in enterprise-level systems.

MariaDB offers great concurrency control and is widely adopted for its high availability, while Oracle SQL is known for handling mission-critical workloads with reliability and efficiency. On the NoSQL side, Cassandra is another powerful option, designed for large, distributed systems where fault tolerance and horizontal scaling are paramount. Companies like Netflix and eBay use Cassandra for its ability to handle massive amounts of data across geographically distributed clusters.

In addition to PostgreSQL, MongoDB, and other large-scale databases, tools like Cassandra, Couchbase, and MariaDB offer tailored solutions depending on the needs of your application. Each database has strengths that make it a good fit for high-traffic, data-intensive applications.

As your application continues to grow and evolve, the decision between SQL and NoSQL will depend on the type of data you're managing and your specific scalability needs.

In the next step, we’ll dive into a different use case: applications that are event-driven and require real-time data processing.

Event-Driven Applications

So far, we’ve talked about small-scale and prototype applications, as well as those requiring data integrity. But another major category that’s increasingly popular in today’s tech landscape is event-driven applications.

Whether it’s real-time notifications, WebSockets, or IoT systems, event-driven architectures are everywhere. These types of applications rely on responding to events as they happen, often in real-time, which is crucial for things like stock trading platforms, real-time analytics, and messaging services.

For event-driven systems, the challenge is not just in processing the data but in doing so at scale, with high-speed delivery and reliability. To handle these demands, you need a combination of tools designed for fast data access, real-time streaming, and reliable storage.

Personal Recommendations

For real-time data handling, Redis is an ideal tool. Redis is an in-memory data store, meaning it’s incredibly fast, making it perfect for caching and managing high-speed data flows.

In an event-driven application, Redis can serve as a temporary, fast-access storage for real-time data that needs to be processed and delivered quickly to multiple clients. For example, in a stock trading platform, Redis can store the latest stock prices, which need to be retrieved in milliseconds to update dashboards or send alerts to users.

When it comes to managing data streams, Kafka or RabbitMQ are excellent choices. Both are messaging systems that handle real-time data pipelines, allowing you to distribute data to multiple listeners as soon as events occur.

For instance, in a trading system, data on market transactions can be streamed through Kafka, ensuring that updates are distributed to various services, such as those that handle client notifications or analytics processing.

But in an event-driven architecture, fast data access and streaming alone aren't enough—you also need to persist the data for long-term storage. This is where databases like MongoDB come in.

MongoDB, with its schema-less design and scalability, can store event data over time, making it available for future queries or analysis. In combination with Redis and Kafka, MongoDB ensures that your system can handle both real-time data and long-term persistence without sacrificing performance.

Example Use Case

Let’s consider a trading system as a use case to see how these tools can work together. In such a system, market data needs to be ingested, processed, and delivered to users as quickly as possible. Here’s how it might work:

Redis stores the real-time price updates for stocks, allowing the application to retrieve this data in milliseconds and send it to clients via WebSockets.
Kafka manages the flow of data from external sources, streaming market data into the system and distributing it to different services, such as pricing engines, notification systems, and analytics modules.
MongoDB stores all the historical transaction data, allowing traders to access past information for analysis or reporting.

By combining these tools, you can build an event-driven system that handles large volumes of real-time data while ensuring fast access and reliable storage.

This approach allows you to scale effortlessly, maintaining high performance even as the number of users and the amount of data grow.

My Thoughts

At this point, I think we’ve covered some of the most common tools that can be used for a wide range of applications you might encounter in your journey as a developer. Naturally, different systems and applications will call for different solutions, and the databases we've discussed are just a few examples.

So, I encourage you to continue your research, dig deeper, and explore other tools that may be a perfect fit for your specific needs.

However, it's important to remember that none of this is set in stone. These recommendations aren’t meant to be absolute truths or universal solutions. The key is choosing the right tools for your project.

For example, if you're working on a small-scale application and you feel confident about using something like PostgreSQL to validate your idea, go for it! Just keep in mind the potential technical debt that might come with overengineering early on. Similarly, if you’re building an event-driven system and decide to rely on a traditional database for real-time data storage, that can be a viable choice—just be aware of the trade-offs that come with that decision.

At the end of the day, it's all about finding the right balance for your system. Always research thoroughly, talk to people who have used these tools in the field, and seek advice from others in your industry. Understand how these technologies fit into your project and what works best for your particular needs.

Conclusion

Thank you for reading! In this article, we’ve explored different databases suited for various types of applications, from small-scale prototypes to data-heavy, event-driven systems.

If you have experiences to share or any thoughts on the article, feel free to drop them in the comments below.

I’m always open to feedback and happy to engage in discussions that can help us all learn and grow.

If you enjoyed this article and want more insights like this, subscribe to my newsletter for weekly tips, tutorials, and stories delivered straight to your inbox!

How to Talk to Non-Developers?

Mangabo Kolawole — Sun, 06 Oct 2024 06:13:08 +0000

Let’s face it—being a developer isn’t just about writing flawless code. It’s about collaboration. But here’s the harsh truth: most developers suck at communicating with non-developers.

What happens when you’re trying to explain something to designers, QA testers, project managers, or marketing professionals? How many times have you seen blank stares or heard the dreaded, “I don’t get it”?

It's not entirely their fault, nor yours, but you can make some efforts to make the communication clear.

In today's article, explore some principles for communicating with non-developers.

Personal Story

In 2020, I worked at my first startup. In the beginning, the team was made up entirely of developers, all working hard to create the MVP.

Two months before the product launch, our first non-technical teammate joined: a marketer.

In a startup, understanding the product is crucial. You might find yourself involved in areas that don’t directly relate to your job.

One day you’re coding, and the next you’re learning about marketing strategies because the founders want your input. That’s why working in a startup is so rewarding—you get to learn a lot.

Our marketer felt the same way. As she got more familiar with the product, she started making suggestions. But when it came time to implement some complex features, she asked me the toughest question:

Ari: "Why?"

Me: "Our WebSocket engine can’t handle that many requests. It’s complicated."

Her confusion was clear. That’s when I realized I needed to explain things in a simpler, clearer way. Thankfully, a more experienced developer stepped in with a better response:

Him: "We have a messaging system that sends notifications to consumers, developers, and riders. Right now, it’s not stable enough to add more recipients. But we could improve things by sending push notifications and listing orders directly in the app, instead of waiting for notifications to show up. What do you think?"

I loved that answer because it gave me a blueprint for how to communicate with non-technical people:

Keep the language simple.
Forget the technical details; focus on the requirements.
Be patient and open to collaboration.

Another colleague of mine—a very technical person—learned a similar lesson. He was the classic “geeky” developer shown in movies and TV shows, always buried in code. In his first few months with us, he had to adjust one major habit: speaking in technical jargon to the manager.

He was a mobile developer, rewriting our app from React Native to Flutter. One day, when he was behind on an implementation, the manager asked why.

Instead of giving a simple, abstract explanation, he dove into details about classes, proxies, and components. As a backend engineer with no knowledge of Flutter architecture, even I was confused. So, you can imagine how lost the manager was.

Luckily, another team member stepped in to save the day. He explained the situation without going too deep into technicalities:

Coworker: "Flutter works differently from what we’ve used before. We assumed a part of the implementation would be the same, but there’s no support for it, so we have to write our own solution. That’s why it’s taking time. We can have it ready by [new date]. Does that sound okay?"

He kept things simple, avoided unnecessary technical details, and shifted the focus to the requirements. He also asked for feedback, which opened the door to collaboration. This made the conversation smoother and more productive.

The Takeaway

Being simple and abstract is key when communicating with non-developers. They don’t need to know the technical complexities behind every issue.

Sharing too much technical detail can make the problem seem more complicated than it is, which might create unnecessary anxiety.

Simplicity is crucial—start with an abstract explanation, and if they ask for more details, you can go deeper.

Redirect the conversation to the requirements and offer solutions or timelines when possible. This helps keep the focus on what matters most to non-technical teammates.

Finally, always invite input. Asking for their thoughts encourages discussion and fosters better collaboration.

At the end of the day, it's not about proving technical expertise—it’s about ensuring that the team can work together to achieve the same goal.

Share your experience in the comments below, ask any questions you have, and don’t forget to share this article with your network if you found it helpful.

If you enjoyed this article and want more insights like this, subscribe to my newsletter for weekly tips, tutorials, and stories delivered straight to your inbox!