DEV Community: Pawel Kadluczka

Building a Computer That’s Older Than the Internet

Pawel Kadluczka — Mon, 05 Jan 2026 03:12:35 +0000

Monty Python is awesome. But there is one thing the Brits gave to the world that is even better: the ZX Spectrum.

Introduced in 1982, the ZX Spectrum was a computer that sparked the home computer revolution in Britain and, due to its affordability, in many other European countries. The basic version had 16 KB of RAM, while the enhanced version boasted an immense 48 KB.

Though I’ve never owned a ZX Spectrum, it is close to my heart. It was the first computer I had a chance to work with. The computer club I was going to as a kid had a dozen of these computers, and we couldn’t wait for the teacher to finally stop talking and let us play some games.

Fast forward 40 years, and while browsing eBay, a brand-new replacement case for a ZX Spectrum showed up in my search results. The memories came back, and I impulse-bought it immediately. I didn’t have any plan for it, but once I got it, I knew I had to put it to good use, i.e., build a ZX Spectrum replica.

Getting started

A nice case is a good start, but what goes inside is key. I found a few interesting options, but the Harlequin 128K Kit from ByteDelight was by far the best. This kit contained everything I needed to build a fully functional ZX Spectrum.

When I received my Harlequin 128K Kit, I felt overwhelmed. While the ZX Spectrum had only a handful of chips (not counting RAM), this kit had more than 50. There is a good reason for this - the kit uses widely available, off-the-shelf chips to emulate the ZX Spectrum’s ULA (Uncommitted Logic Array) specialized chip that went out of production almost 40 years ago.

I had never done much soldering, and the 50+ chips alone meant hundreds of solder joints. But there were also resistors, diodes, transistors, etc. The website promised it would be a fun challenge and that basic skills and creativity were enough to pull it off. These were encouraging words, but what the website didn’t mention was that a single mistake would ruin it all.

To make the assembly process easier, the kit parts were divided into 40 or so clearly labeled bags and provided with the recommended installation order. The detailed assembly guide included additional hints and called out easy-to-miss gotchas. Given my skill level, I found the kit's organization extremely helpful.

First Try

The instructions said the soldering would take a few hours. Indeed, I spent a few hours a day for a week building the board. I followed the assembly steps very closely because I knew troubleshooting any non-obvious issues would be beyond my abilities. Eventually, I emerged from my garage with the board, anxious to try it.

I connected it, and what I saw made my heart sink.

If the board had been completely dead, I might have hoped I had missed a connection, but what I saw indicated a subtle issue I doubted I could fix. A quick Internet search only confirmed my suspicion. A few people reported similar symptoms, but no one could propose a reliable fix.

I verified that all the chips were inserted correctly and pressed them down firmly to ensure good contact. I inspected my soldering but didn’t find any missed solder joints. I double-checked the jumpers and found that a couple of them controlling the video settings were misconfigured. I corrected them, but nothing changed. Frustrated, I gave up.

Round 2

The problem haunted me. A week later, I went over everything again and confirmed I hadn’t missed anything. As I was out of ideas, I started playing with jumpers for ROM selection. To my astonishment, one of my changes produced this screen:

This meant the board had been working fine all along! It was just trying to run some garbage instead of a valid ROM program.

ROM

I turned to the guide to understand what was happening. Due to copyright, the kit doesn’t ship with the original ZX Spectrum ROM. Instead, it comes with the AM29F040B EEPROM, which is big enough to hold up to 8 different ROM images. The board uses jumpers to select the image to run. Apparently, some of the banks in my EEPROM didn’t have a valid ROM image, and my initial configuration happened to point to one of these banks.

Now that I understood the issue, the fix was easy: I needed to burn the ZX Spectrum ROM into my EEPROM. The only problem was that I didn’t have an EEPROM programmer. After some research, I settled on the XGecu T48 EEPROM programmer. When I got it, I was unpleasantly surprised: the official software only supported Windows, but I am a Mac user.

I sacrificed a lot for this project, but having to install Parallels would be too much. Fortunately, I found an open-source command-line tool called minipro that offered what I needed.

The Harlequin 128K board I built was compatible with both ZX Spectrum and ZX Spectrum 128. Because my EEPROM could hold multiple ROM images, I burned them both so I could easily switch between them.

Visual Minipro

While the minipro tool did the job, setting it up and understanding the options took some effort. A UI version of the tool would make everything so much easier. As nothing like this existed, I decided to build it myself. Because my tool is a minipro GUI wrapper, I called it Visual Minipro and made it available on the Mac App Store.

Finishing the build

I was delighted to see my board up and running, but two problems bothered me: the video quality was pretty bad, and I couldn’t use a joystick.

Video

The Harlequin 128K board offers two video outputs: composite and RGB. The RGB signal is intended for use with SCART, which has never been a thing in the US. Since none of my monitors supported SCART, composite video felt like a better choice. Besides, exposing the RGB socket would require cutting a hole in my case, which I didn’t want to do.

To get NTSC timings over composite video, I installed the NTSC crystal that came with the kit. It did the trick, but the result wasn’t great. The picture on my 1084S monitor was blurry, and when I tried to convert it to HDMI using my Retrotink 2X Pro converter, I got no color.

Joystick

In the 8-bit era, a joystick was a must-have device. Most games couldn’t even be played without a joystick. While the ZX Spectrum didn’t support a joystick out of the box, third-party joystick interfaces quickly filled the gap. The most popular one was the Kempston joystick interface.

The Harlequin 128K board has built-in support for the Kempston joystick interface. This is nice, but there are two problems: the ZX Spectrum case lacks a factory-made hole for a joystick port, and the board lacks mounting points for the DB9 joystick socket. So, even if I had modified the case, that case might not have been strong enough to provide solid support for the joystick port. Besides, I knew the DIY joystick hole would look ugly. So, I decided to look for something different.

ZX VGA Joy to the rescue

One popular way to get HDMI output on the ZX Spectrum is ZX-HD, an HDMI interface for the ZX Spectrum. It is a fine device, but it wouldn’t solve my joystick problem. So, I kept looking for alternatives and found something that fit my needs perfectly: ZX VGA Joy. This device could output digital video over HDMI and provided a joystick port compatible with the Kempston interface. As a bonus, it also had the reset button.

I ordered it, and couldn’t wait to try it. When it finally arrived, I wasn’t able to make it work with my setup reliably - the screen would go black soon after turning the Spectrum on. Quick debugging proved this was an issue with the computer - the composite video worked just fine. After reading more about the inner workings of ZX VGA Joy, I got a hunch about what was happening. ZX VGA Joy relies on PAL timings to generate the HDMI output. But my computer had the NTSC crystal installed. My hypothesis was that this difference was responsible for the glitch because the PAL and NTSC frequencies differ.

My guess was correct. Replacing the NTSC crystal with a PAL crystal indeed fixed the blank-screen problem, but it came at a cost. After installing the PAL crystal, I lost the ability to use the composite video. This wasn't a big deal, though. Given the low quality of the composite signal, I wouldn’t want to use it anyway.

Final result

Here is the final result after putting everything together (with a modded Nintendo NES controller in lieu of a joystick and the original SimCity game humming happily on 48KB of RAM).

Conclusion

There is no practical reason to build a 40-plus-year-old computer today. To be honest, even when I was starting, I didn’t expect to really use it. However, the process of building it, the interesting technical challenges, and the satisfaction of seeing it come to life were all well worth it.

Using mental models to think about software

Pawel Kadluczka — Sat, 23 Nov 2024 02:31:27 +0000

No person can function properly without building mental models to understand the world around them. The reality is simply too complex to deal with as is, so we need to abstract it.

Software is no different. The complexity of software systems has been only growing. As a result, the ability to quickly build mental models and use them to reason about these systems is an important software engineering skill.

Building good mental models is important because they allow us to better communicate with other software engineers, explain our ideas, and understand the impact of our changes on our software.

Building mental models

Reasoning about software systems requires building mental models that represent them. The granularity of these models depends on their purpose. Often, one level of granularity is not sufficient. The most effective software engineers can quickly build a few models and switch between them depending on the situation.

For example, if your team owns a few services, you can draw a lines-and-boxes diagram where boxes represent services and lines represent dependencies. You can then zoom in and build a model for each service. These service-level models could illustrate interactions between the libraries a service consists of. If you want more details, you can create a class diagram. You can continue zooming in and focus on methods, code blocks, statements, etc.

Each of these models describes your system at a different level of granularity and has a unique set of applications. The high-level model could be useful to troubleshoot larger outages (or when talking with your director) but is unlikely to help you fix a small bug. The more detailed models are best suited for solving gnarly issues but won't be helpful when explaining your infrastructure to other teams.

Building models covering different aspects of the same system is also common. If you want to analyze your system from the security perspective, your model will include different details than when focusing on performance.

Caveats

Mental models are so natural to people that we often forget about their flaws.

All models are wrong

Models, by definition, ignore details. As they only capture certain aspects of reality, they are inaccurate. Furthermore, some relevant information is often omitted because it doesn't fit the model. Edge cases are an excellent example.

For instance, when software developers explain how their code works, they rarely mention error cases. They focus on their *if*s and *for*s and the program flow but omit exceptions because exceptions make their model murkier. The problem is that error scenarios are important. Incorrect or missing error handling is a common cause of system outages.

Models get more wrong with time

The world, including our software systems, is in constant flux. But mental models don't automatically keep up with changing reality. Outdated mental models lead to misunderstandings and bad decisions.

I experienced this very problem recently. I started working on a feature that depended on a system I had never seen changing. Everything was going swimmingly, and I only needed to tie up a few loose ends related to some new data requirements. But when doing this, I discovered, to my dismay, that the system I depended on had recently changed. It had been updated to accommodate the same data requirements I struggled with. Making my feature work now required additional information I didn't have. Plumbing this data meant revisiting my implementation. Working off of an outdated mental model cost me implementing my feature twice.

No two models are identical

Building mental models requires deciding which details are important depending on the purpose of the model. However, even if the purpose of the model is well understood, different people will consider different information relevant.

Also, mental models built at different times will naturally differ because they capture different system versions.

The interesting fact about this phenomenon is that the overlap between mental models built by different people is usually significant. The differences are often discovered unexpectedly, e.g., when discussing small but important details.

💙 If you liked this article...

I publish a weekly newsletter for software engineers who want to grow their careers. I share mistakes I’ve made and lessons I’ve learned over the past 20 years as a software engineer.

Overthinking Software Projects

Pawel Kadluczka — Sat, 16 Nov 2024 06:40:17 +0000

"Weeks of coding can save you hours of planning."

Most developers learn this truth the hard way. I did when I implemented a feature based on incorrect assumptions, and the only way to save it was to rewrite it.

While junior engineers often fall into the trap of jumping straight to coding without thinking about the problem, more senior engineers fall into a different trap: overthinking.

Senior developers are routinely responsible for projects requiring a handful of engineers and a few months to complete. There is no way to execute these projects without careful thinking and planning. Even if you wanted to start coding on the first day, you couldn't. You simply wouldn't know where to start.

The top priority in the initial phases of bigger software projects is to sort out the ambiguity that blocks development. The most common way to do this is to devise a design that will guide the implementation.

The tricky part is that figuring out the design is by itself an ambiguous problem. There is usually more than one way to implement the solution, and many constraints and requirements are unclear. Even estimating how long it will take to prepare the design can be difficult.

All these uncertainties put pressure on the engineer(s) responsible for the project. Making a bad decision can lead to wasted time (amplified by the size of the team) or even a project failure.

When stakes are high, it is natural to proceed carefully, explore potential problems, and work with others to identify gaps that may otherwise go unnoticed. However, setting limits for these activities is crucial. Failing to do so will inevitably lead to overthinking, which can also put the project at risk due to:

Delays - analyzing every imaginable scenario takes a long time and will eat into development time, making meeting expected timelines impossible.
Overengineering - trying to address minor or hypothetical issues leads to overly complex designs that are hard to implement and expensive to maintain.
Missed opportunity - endless discussions, revisions, and feedback rounds steal time engineers could spend on other project activities or elsewhere.

How do you know if you are overthinking?

One of the problems with overthinking is that the line between productive analysis and overthinking is thin and easy to miss. However, there are signs of getting there:

Although all critical requirements have been satisfied, new requirements are being added.
The same topics continue to be discussed repeatedly without reaching any resolution.
Edge cases, minor issues, and esoteric scenarios start to dominate the discussion.
The debate moves to future scenarios that are out of the scope of the project at hand.

What to deal with overthinking?

It is important to understand that the goal of the planning and design phases is not to identify and solve all possible problems. First, it is impossible even to list all the issues. No matter how much time you spend thinking you will miss something. Second, many problems will never materialize, or if they do, their impact will be minimal. To focus on what's important, create a list of requirements, decide which are critical (the list should be short), and satisfy those. Leave out the remaining ones and revisit them if they become a major problem.

Many problems have no one correct or even best solution. No amount of debate is going to change that. It may take time to arrive at this conclusion, but once it is settled, you must pick one of the alternatives and live with the consequences.

If you are not sure, prototype. An hour of coding can save you a few hours of meetings. Code does win arguments.

Design for your current needs. If you don't operate at Facebook's scale, don't design for Facebook's scale. Don't build abstractions for your hypothetical future features.

Accept the fact that things may not work out. Fortunately, you are not pouring concrete. This is software, and the ability to modify it is one of its greatest advantages. You can build small and evolve your solution when your needs grow.

Build vertically. A small feature working end-to-end will allow you to discover issues early and course-correct. If you build horizontally, you won't see gaps until very late when all the layers are ready. Fixing bigger problems at this stage will be an undertaking.

Picture: The “Am I Overthinking This?” book cover.

💙 If you liked this article...

I publish a weekly newsletter for software engineers who want to grow their careers. I share mistakes I’ve made and lessons I’ve learned over the past 20 years as a software engineer.

How to quickly ramp up on new codebases

Pawel Kadluczka — Fri, 08 Nov 2024 06:55:44 +0000

Joining a new team is intimidating. There is always much to learn - new people, new processes, and a new code base.

Ramping up on your new team’s code base is difficult, but knowing how to work effectively in it is crucial to your success. So, how do you do it quickly?

My mid- and early senior developer years were intense. Due to a mix of reorgs and personal interests, I found myself on a new team every year or so. As a result, I had to learn new codebases in quick succession. They included .NET System.Xml, OData, Entity Framework, Entity Framework Designer, ASP.Net SignalR, ASP.Net Core, and the Alexa mobile app, and most of them were over one hundred thousand lines.

The first couple of transitions were slow and overwhelming. But they helped me develop a strategy I used later to quickly get up to speed on new codebases.

Get your hands dirty ASAP

The sooner you start working actively with the code base, the sooner you will get productive. Reading documentation can be helpful, but nothing can replace the hands-on experience.

When I joined Amazon, I asked my manager to assign me a simple bug to fix on my first day. Even though the fix amounted to a single line of code, it took me a few days to send it for review. It might seem long, but this quest was not about fixing the bug. Rather, it was about forcing myself to set up my development environment, learn how to build and run the code, write unit tests, and debug our product.

Review code

Each team has its way of letting members know there is a new PR (Pull Request) for review. It could be email, chat, or review tool notifications. Whatever it is, subscribe to this channel and start reviewing PRs. You won’t understand much initially, but you may still catch some bugs (e.g., off-by-one errors). More importantly, these reviews will allow you to ask questions and gather a broader context of what the team is working on.

Identify code that matters

The 80/20 rule does apply to codebases. This is especially noticeable in the bigger ones, where most code changes developers make are concentrated in one area. Knowing which code it is allows you to focus your ramp-up on the area you are likely to work on soon.

There are a couple of easy ways to tell which code is in the top 20%:

paying attention to code reviews
checking the commit history

Take notes, draw diagrams

I discovered that taking notes and drawing diagrams is a very effective way to grok the most complex parts of the code. Call graphs, class diagrams, and dependency graphs all help organize the information and are great reference material in case you need to refresh your memory.

After I joined Amazon, I drew diagrams of a few areas of code I couldn’t understand. They became an instant hit. Even people who had been on the team for much longer than me wanted a copy.

Debug code

Reading a new codebase is like reading a book in a language you barely know. You can do it, but it is excruciating.

For me, one of the best ways to overcome this is to step through the code with the debugger. Initially, I have no idea what I am looking at. But if I follow the same code path a few times, I begin to recognize code I have already seen, and soon, everything starts to fall into place.

To get the most out of my debugging sessions, I do two things:

I continuously inspect the state - the stack trace, parameters, and local and class variables
I take notes and draw diagrams

Using the debugger to learn the code base is slower and narrower than reading the code. But it is also much deeper. In fact, when debugging, I often realize that the understanding I gained from reading the code was incomplete or sometimes even incorrect.

Read documentation

Reading documentation can help accelerate your onboarding. It could be especially useful when it comes to the high-level architecture and concepts that are hard to deduce from code.

However, you should take documentation with a grain of salt. It is often sparse and outdated. But this could be good news for you. Updating the documentation as part of your ramp-up could be a great contribution to your new team.

Join on-call rotation

Joining the on-call rotation to accelerate team onboarding might sound extreme but I did it on my first team at Facebook. I did it because I was concerned that my ramp-up was slow, so I decided to push myself. I learned more about our services this week than in weeks before. After my shift ended, I knew what services our team owned, their dependencies and where to find their code. Alerts immediately pointed me to the hot code paths, and troubleshooting issues forced me to dig into the code.

💙 If you liked this article...

I publish a weekly newsletter for software engineers who want to grow their careers. I share mistakes I’ve made and lessons I’ve learned over the past 20 years as a software engineer.

The Must-Have Skill Every Senior Developer Needs

Pawel Kadluczka — Sat, 02 Nov 2024 17:36:37 +0000

Writing code is a fundamental skill every junior software developer needs to master. However, coding skills are no longer the biggest differentiator at senior and above levels. Every senior engineer is expected to have solid coding skills, and growing to higher levels based on coding alone is rare.

If not coding, then what?

If coding is not the skill to grow beyond senior levels, what skill is it?

This question has no correct answer, as no single skill can elevate you to the staff+ levels.

However, software development is a team sport, and successful senior engineers must focus on many areas besides coding. They are often responsible for projects spanning one or more teams. They drive the design, collaborate with partner teams, communicate progress, etc. Doing all this work effectively requires good communication skills, especially writing (which I consider one of the Universal Skills.)

Why writing?

Writing clarifies thinking and promotes the exploration of ideas. I don't know how many times I thought I understood something, only to struggle to summarize it in writing. But once I succeeded, I had a much deeper grasp of the concept and noticed new insights I hadn't considered before.

Thanks to its durability and asynchronous nature, writing is also a great way to scale. You can write something once and refer to it later. Your readers can benefit from it even if you are not around. Here are a few examples from the software engineering field:

Project execution plans are useful for aligning all interested parties: the team that will execute the project, partner teams, your manager, etc., without having to talk to them individually.
Documentation helps avoid explaining the same concepts again and again. It protects the team from scrambling when a key team member leaves the project or the team (see also: bus factor)
Design documents allow for gathering feedback without holding a meeting for all interested parties. They are also an invaluable resource to understand why certain design choices were made and what alternatives were considered.

Writing is difficult

Writing is not natural for most people. Making the content clear, concise, and well-organized is grueling work.

I often see software developers dismay when I ask them to write a rollout plan or a design doc. Some tell me they were relieved to graduate from college because it meant they would never have to write again, and I am shuttering their world.

There are also other reasons why writing is difficult. Many developers have to write in a non-native language. But even native speakers often struggle because the way of writing they learned at school does not serve them at work.

Opportunities to practice writing.

Even though writing becomes important gradually, it doesn't mean you should wait to improve it. On the contrary, the sooner you start, the better. Fortunately, every developer has plenty of opportunities to practice writing on the job.

Emails

Emails are everyone's bread and butter these days. However, many emails are hard to read and understand and, as a result, fail to achieve their goal.

In my first job, our manager asked us to send a weekly email summarizing what we worked on and accomplished in the past week. I was proud of my reports: they were very detailed and explained everything. Despite these emails, my manager kept asking me what I had been working on. When I saw one of these emails years later, I understood. He never read them. I couldn't blame him - it was an unbearable wall of text.

Memos / Announcements

Posts, memos, and announcements meant for a wide audience need to be tailored to that audience. Otherwise, readers won't understand them and will give up reading them.

I recently read a post from my co-worker reporting on the status of our project. The audience of this post was broad (more than 150 people) and included managers, directors, and partner teams. The technical details in this post left me lost despite my heavy involvement in this project. I can only guess what others took away from this post.

Design documents

Good design documents explain complex topics using simple language. This combination makes them hard to write, but the payoff is worth the effort. Confusing design documents lead to lengthy discussions, feedback on unimportant matters, and frustration.

I once asked a junior engineer to write a design document explaining how he plans to implement a feature we promised to deliver. What I got was an untitled Google Doc with no text and two pictures - a diagram and "The Starry Night" by van Gogh. While I have nothing against "The Starry Night", the document didn't give me the faintest idea about the design of the feature, assumptions, and considered alternatives.

Code review feedback

The main purpose of sending code for review is to gather feedback. But giving short, clear, and actionable feedback professionally is an art. The conclusion: if you want to improve your writing, you should review a lot of code (and provide feedback).

Code comments

I am not a huge fan of writing code comments, but in some situations, they are warranted. Unfortunately, many code comments are so poorly written that it is sometimes hard to tell if they are there to help you or make you more confused.

The main challenge with code comments is that they need to be short to not overshadow the code but must clearly explain intricate ideas that the code cannot express. These requirements make writing code comments good practice.

Documentation

Writing documentation is one of the least favorite tasks software developers want to do. Yet, it often is one of the most impactful they can do. Good documentation helps put out on-call fires faster, makes onboarding new team members easier, and reduces randomization caused by repeatedly answering the same questions. By writing documentation, you help your team achieve more and polish your writing skills.

Bug reports

If you want someone to do something for you, you need to make it as easy as possible for them to do it. If you don't, what you are asking for will take a long time or will never get done.

This rule applies perfectly to bug reports. If you encounter a bug that blocks your work, writing a clear bug report dramatically increases the chances of getting the issue fixed. Despite this, many reported bugs are incomprehensible.

At Microsoft, I worked on a few high-profile open-source projects like Entity Framework or ASP.Net Core. As thousands of developers used our products, we received a decent number of bug reports. Unfortunately, we often couldn't understand what issue was being reported, how to reproduce it, and the expected behavior. Following up on these issues was painful. The back-and-forth took weeks. The "bugs" slipped from release to release while we were waiting for the details we requested. Eventually, we closed most of these bugs without resolution as it was hard to prioritize them over other issues we could immediately investigate and fix.

💙 If you liked this article...

I publish a weekly newsletter for software engineers who want to grow their careers. I share mistakes I’ve made and lessons I’ve learned over the past 20 years as a software engineer.

Understand the purpose of your work

Pawel Kadluczka — Sat, 26 Oct 2024 18:20:06 +0000

One mistake I've seen junior software engineers repeat again and again is their lack of understanding of why they work on tasks they work on. This confusion can be somewhat justified by the relatively small scope junior engineers typically have but it's a slippery slope. Doing something only "because my manager (or a senior engineer) asked me to do it" has a few drawbacks:

Inability to execute independently: making even the smallest decision without involving your manager or the tech lead will be hard if you don't understand the bigger picture. You will get stuck if you can't get hold of them. It will also be difficult for you to demonstrate you know how to solve problems at your level and are ready for bigger challenges.
Communication gap: your manager or senior engineer may unintentionally give you incomplete or incorrect information. If you don't have enough context, you may not notice this. You may struggle to complete the task, but once you finally do, it may turn out that what you built is not what they hoped for and needs redone.
Hindered innovation: unawareness of where your work fits limits your ability to propose solutions beyond what you're asked to do. Sometimes, the approach you're instructed to follow may not be the best solution to the problem, but exploring alternatives is impossible if you don't understand the broader context.
Incorrect prioritization: working on a task without knowing its purpose may lead to neglecting this task and unknowingly delaying work that depends on it.

How to understand the bigger picture?

The easiest way to understand where your work fits is to ask your manager or the tech lead. They are responsible for what the team needs to deliver, so they should be able to explain this instantly.

Your question may even come to them as a surprise. They probably assume everyone on the team already understands the purpose of their work. In my experience, this is not always the case. The bigger and more complex the project, the harder it is to connect the dots.

You can start small, but it is important to go deep. Start asking about your task. You may hear that it contributes to a project the team is working on. An answer like this is not very helpful but could be a great starting point. It allows you to drive the discussion further and ask more interesting questions like:

Why are we working on this project? Why is it important?
What metrics is this work expected to move, and how?
How does it support the company's goals and priorities?
What projects did we decide not to pursue to fund this work (a.k.a. opportunity cost)?

A different way to understand where your work fits might be by talking to your product manager or people from the UX (User Experience) or marketing team. Because of their different perspective, they can teach you things you would never learn from fellow engineers. The challenge with this approach is that you need to be able to explain your role in the project to them.

💙 If you liked this article...

I publish a weekly newsletter for software engineers who want to grow their careers. I share mistakes I’ve made and lessons I’ve learned over the past 20 years as a software engineer.

Want your productivity to skyrocket? Avoid this trap!

Pawel Kadluczka — Sat, 19 Oct 2024 04:49:42 +0000

As a junior engineer, I felt the urge to jump on each new project that showed on the horizon. I never checked what I already had on my plate. Inevitably, I ended up with too many projects to work on simultaneously. Whenever my manager or a customer mentioned one of my projects, I immediately switched to it to show I was making progress. It took me years to realize that while this approach pleased the customer or the manager (and saved my junior dev butt) for the moment, it quietly hurt everyone.

The three-line execution graph

Executing multiple projects of the same priority at the same time looks like this:

Initially, there are two projects: Project A and Project B. You start working on Project A, but after a while, you receive a call from the customer inquiring about the progress of Project B. To make this customer happy, you switch to project B. In the meantime, a new interesting project, C, pops up. It is cool and seems small, so you pick it up. Your manager realizes that project A is dragging and asks about it. You somehow manage to finish your toy project C and move to A, which is well past the deadline. Then you pick B again.

If you didn't jump from project to project, the execution could look like this:

If you compare these graphs, three things stand out in the second scenario:

Overall, the execution took less time. Resuming a paused project requires time to remember where the project was left off and get in the groove, i.e., to switch the context, which , which is time-consuming.
Projects A and B finished much quicker than in the first case. While you didn't make customer B feel good by saying you were working on their project, ultimately, the project was completed sooner. In fact, both projects, A and B, were finished much sooner than they would have if you bounced between them.
Project C came in late, so it should wait unless it is a much higher priority than other projects. Otherwise, it disrupts the execution of these projects.

I know that life is not that simple. Completely avoiding context switching is rarely possible. But if you can limit it, your productivity will dramatically increase.

Does it mean you should only work on one thing at a time?

In the past I thought a good solution for junior engineers to combat context switching was to ask them to work on just one project at a time. But this idea has a serious drawback - projects often get stuck due to factors outside our control. If this happens and there is no backup, idling until the issue gets resolved is a waste of time.

Having two projects with different priorities works best. You execute on the higher-priority project whenever you can. If you can't, you turn to the other project until the main project gets unblocked.

What I like about this approach is that it is always clear what to work on: the higher priority project wins unless working on it is impossible.

Falling back to the lower priority project means there might be some context switching. While it is not ideal, it is better than idly waiting until the issues blocking the main project are resolved.

But my TL (Tech Lead) always works on five projects!

Indeed, experienced senior and staff engineers often work on a few projects at the same time. In my experience, however, it is a different kind of work. It might be preparing a high-level design, working on an alignment with a partner team, breaking projects into smaller tasks, and tracking the progress.

The secret is that most of these activities don't require as much focus as coding. Handling a few of them at the same time is much more manageable because the cost of switching contexts is much lower.

💙 If you liked this article...

I publish a weekly newsletter for software engineers who want to grow their careers. I share mistakes I’ve made and lessons I’ve learned over the past 20 years as a software engineer.

A simple way to ship maintainable software

Pawel Kadluczka — Fri, 11 Oct 2024 00:20:42 +0000

This was my first solo on-call shift on my new team. I was almost ready to go home when a Critical alert fired. I acknowledged it almost instantly and started troubleshooting. But this was not going well. Wherever I turned, I hit a roadblock. The alert runbook was empty. The dashboards didn't work. And I couldn't see any logs because logging was disabled.

Some team members were still around, and I turned to them for help. I learned that the impacted service shipped merely a week before, and barely anyone knew how it worked. The person who wrote and shipped it was on sick leave.

It took us a few hours to figure out what was happening and to mitigate the outage. This work made one thing apparent - this service was not ready for the prime time.

In the week following the incident, we filled the gaps we had found during the outage. Our main goal was to ensure that future on-calls wouldn't have to scramble when encountering issues with this service.

But the bigger question left unanswered was: how can we avoid similar issues with any new service or feature we will ship in the future?

The idea we came up with was the Service Readiness Checklist.

What is the Service Readiness Checklist?

The Readiness Checklist is a checklist that contains requirements each service (or a bigger feature) needs to meet to be considered ready to ship. It serves two purposes:

to guarantee that none of the aspects related to operating the service have been forgotten
to make it clear who is responsible for ensuring that requirements have been reviewed and met

When we are close to shipping, we create a task that contains a copy of the readiness checklist and assign it to the engineer driving the project. They become responsible for ensuring all requirements on the checklist.

Having one engineer responsible for the checklist helps avoid situations where some requirements fall through the cracks because everyone thought someone else was taking care of them. The primary job of this engineer is to ensure all checkboxes are checked. They may do the work themselves if they choose to or assign items to people involved in the project and coordinate the work.

Occasionally, the checklist owner may decide that some requirements are inapplicable. For example, the checklist may call for setting up deployment, but there is nothing to do if the existing deployment infrastructure automatically covers it.

The checklist will usually contain more than ten requirements. They are all obvious, but it is easy to miss some just because of how many there are.

Example readiness checklist

There is no single readiness checklist that would work for every team because each team operates differently. They all follow different processes and have their own ways of running their code and detecting and troubleshooting outages. There is, however, a common subset of requirements that can be a starting point for a team-specific readiness checklist:

[ ] Has the service/feature been introduced to the on-call?
[ ] Has sufficient documentation been created for the service? Does it contain information about dependencies, including the on-calls who own them?
[ ] Does the service have working dashboards?
[ ] Have alerts been created and tested?
[ ] Does the service/feature have runbooks (a.k.a. playbooks)?
[ ] Has the service been load tested?
[ ] Is logging for the service/feature enabled at the appropriate level?
[ ] Is automated deployment configured?
[ ] Does the service/feature have sufficient test coverage?
[ ] Has a rollout plan been developed?

Success story

Our team was tasked to solve a relatively big problem on tight timelines. The solution required building a pipeline of a few services. Because we didn't have enough people to implement this infrastructure within the allotted amount of time, we asked for help. Soon after, a few engineers temporarily joined our team. We were worried, however, that this partnership may not work out because of the differences in our engineering cultures. The Service Readiness Checklist was one of the things (others included coding guidelines, interface-based programming, etc.) that helped set clear expectations. With both teams on the same page, the collaboration was smooth, and we shipped the project on time.

💙 If you liked this article...

I publish a weekly newsletter for software engineers who want to grow their careers. I share mistakes I’ve made and lessons I’ve learned over the past 20 years as a software engineer.

The 3 categories of skills every software developer needs to know

Pawel Kadluczka — Fri, 04 Oct 2024 06:06:04 +0000

100% of software engineers who don't keep their skills sharp become obsolete[*]. Many who do keep their skills sharp also become obsolete. It all boils down to what skills they focus on.

I divide skills into three main categories: company-specific, job-specific, and universal. This categorization makes it easy for me to decide where to invest my time when it comes to skill development.

Company-specific skills

Each company has internal infrastructure, processes, and tools. Knowing them before joining the company is impossible, but everyone must learn them after joining.

Being strategic about what to focus on can save you a lot of time. Company-specific knowledge and skills are, by definition, not transferable. You need to learn them to do your job, but they become useless the moment you move on.

My strategy is to have a solid understanding of company-specific tools and processes but not dive too deeply unless I have to. Being in the dark will slow me down, but drilling into all the details is not much better. Most of these things constantly change, and I will quickly forget what I don't use. Instead, my time is better spent developing other, more general skills.

When I worked at Microsoft, I had a colleague named Ben. Ben was the most knowledgeable person I knew when it came to the .NET Framework build system. Although he was not on the build team, he knew all the scripts, hacks, and environment variables used in this build system. Acquiring and keeping this knowledge fresh ate much of Ben's time. Having Ben around was great for the team. We could (and constantly did) ping him for help with build-related issues. Eventually, Ben moved to a different team outside our organization. His expertise evaporated instantly, even though he didn't go far.

Job-specific skills

Job-specific skills are usually the skills that get us hired. Companies look for people who can hit the ground running, and having skills in demand increases the chances of getting hired substantially.

Job-specific skills are transferable. You learned them before getting hired and may use them in your next job. There is a caveat, though. It is only true if you keep your skills up-to-date. Knowing React and knowing React-as-of-four-years-ago is not the same thing.

I learned that trying to keep skills sharp just by using them at work doesn't always work. Companies rarely move as fast as technology. For instance, the product you work on might have been on the bleeding edge a few years ago. But it got stuck there because there was never a good enough business reason to migrate it to the newest framework version. You still need to maintain it but can't use the latest features.

Another thing to pay attention to is the signs that the technology you specialize in is becoming obsolete. Some technologies are more durable, but some can fade fast. You don't want to wake up one day only to realize that everyone except you has moved on.

CoffeeScript was one of the most followed projects on GitHub in the early to mid-2010s. It was incredibly popular, and there was a lot of hype around it. Today, hardly anyone remembers it.

Even if you got hired for your job-specific skills, it doesn't mean you can't learn new skills on the job. If you want to develop a new skill, you may consider joining a new project or moving to a different team. It isn't always easy because you don't have the skills they need, but it is doable - especially if you are known as someone who learns fast.

And the biggest secret: not only do you develop a new skill that may land you your next job, but you are also paid to do this.

Universal skills

Finally, there are the universal skills. These are skills that never become obsolete. You can use them at your current job, at your next job, or for non-work-related purposes. You can also apply them instantly.

These are skills like writing, effective time management, delivering great presentations, etc.

The biggest problem with universal skills is that developing them never seems urgent. As a software developer, you will not lose your job because you write lengthy emails. But eventually, you may hit a glass ceiling and realize that what inhibits you is not coding but the lack of these universal skills.

--
* - citation needed

💙 If you liked this article...

I publish a weekly newsletter for software engineers who want to grow their careers. I share mistakes I’ve made and lessons I’ve learned over the past 20 years as a software engineer.

These 5 habits will make you a great code reviewer

Pawel Kadluczka — Fri, 27 Sep 2024 14:47:57 +0000

High-quality code reviews are hard.

They are time-consuming and require significant mental effort.

Code reviewing is also not taught at school, and figuring it out on your own is hard work. In this post, I share five crucial habits all great code reviewers I know have in common.

Make the code review about the code

Code reviews are about the code, not the person who wrote the code. If you think the proposed change is incorrect or have suggestions to improve it, then, by all means, leave your comments (just remember to make them professional and high-quality). However, leave out comments that don’t relate to the code under review.

Understand the code and ask questions when in doubt

Understanding the code under review is the foundation of a solid code review.

It is also the most difficult part.

Whenever you have doubts about proposed changes, you should ask the author for clarification. You might not be aware of an assumption the author is making or need help understanding how their changes fit in. However, the difficulty in grasping the changes is also often a sign of a mistake.

Asking a question will prompt the author to explain their thought process. As a result, they will either answer the question and clear up your doubts or realize that something is indeed wrong and needs to be fixed.

Be clear about your expectations

Code reviews can generate a wide variety of comments. Some are nits that you would like to see fixed but are not real issues. Some, however, identify serious problems that must be corrected before merging. If you leave a comment, make sure that the author understands which category it falls into. Doing this will save you and the PR author time and frustration.

Sometimes, you may take on a PR that is outside of your area of expertise. You may realize it only after you already left some comments. You are now in a weird situation: the author expects you to finish the review and approve the PR, but you don’t feel confident you can. If this happens, instead of accepting the change you don’t understand, it is better to leave a comment recommending the author get a review from someone more familiar with the code they are changing.

Side note: reviewing code that is outside of your area of expertise is a good thing. Even if you don’t understand the change enough to approve it, you can still provide useful feedback, identify bugs, and learn something. Just make sure the author does not expect to get your approval.

Cross-check with the existing code

Code reviews show only code that changed. Most of the time, it is sufficient. But sometimes, to understand the change fully, you need to check how the new code works with the code not included in the review because it hasn’t changed.

This idea may be obvious to most developers, but I was surprised to meet some who had never considered checking the existing code when reviewing PRs. In my experience, the biggest surprises are caused by not what’s included in the PR but by what’s missing.

Occasionally, examining the existing code may not give you all the answers. The ultimate weapon for these situations is to use a debugger. You can check out the PR branch and step through the code. It should resolve all your doubts. I resort to a debugger very rarely. It’s almost always easier and faster to ask the author.

Resist the “stamp” pressure

The pressure to merge changes quickly for projects on tight timelines is high. And it only grows as the deadline nears.

During the end game, engineers enter a Pull Request frenzy. Eventually, due to the number of PRs, code reviews often become a bottleneck. So, the engineers try to unblock themselves by asking to “stamp the diff” (i.e., approve changes without looking).

On the one hand it is understandable - no one wants to miss the deadline. On the other hand, these are the times when code changes need even more scrutiny than usual. Due to the time pressure, most PRs are coded very hastily. The changes may not be validated thoroughly (or at all) and the stress only increases the likelihood of mistakes.

While a proper code review takes time, merging code with issues a review could have caught is more costly. At best, fixing the problem will require sending a new PR (which, by the way, will trigger a review). At worst, an embarrassing bug ships to customers.

I remember when one of my teams worked extremely hard to finish a project on time. We were very close, and then one of the team members dropped a 1000-line PR a few hours before the deadline. The manager was trying hard to find someone willing to approve the PR. It wasn’t easy because the PR had a bunch of red flags, like spotty test coverage or many TODO comments, but he eventually succeeded. As soon as our product shipped, we started getting reports from angry customers complaining that important scenarios stopped working. We found that the hastily merged PR was the culprit. The team scrambled to fix the issues, but the damage was done. This one PR cost us the reputation we’d been building for a long time.

💙 If you liked this article...

I publish a weekly newsletter for software engineers who want to grow their careers. I share mistakes I’ve made and lessons I’ve learned over the past 20 years as a software engineer.

On-call Manual: How to cope with on-call anxiety

Pawel Kadluczka — Fri, 20 Sep 2024 05:51:53 +0000

On-call anxiety is real. I've been there, and I know engineers who experienced it. Many factors contribute to it, but from my experience, three stand out.

1. Unpredictability

Unpredictability is the number one reason for on-call anxiety. You might be responsible for a wide range of services. They may break anytime for various reasons like network issues, deployments, failing dependencies, shared infra outages, data center drains (a.k.a. storms), excavators damaging etc. On-calls, especially new ones, worry they won't know what to do if they get an alert. How would they figure out what broke? How would they come up with a fix?

What to do about it?

With experience, the unpredictability aspect of the on-call gets easier. But even for the most seasoned on-call engineers, handling an outage can be difficult without the proper tools like:

Easy-to-navigate dashboards that allow to tell quickly if a service is working correctly and identify problematic areas in case of failures
Playbooks (a.k.a. runbooks) explaining troubleshooting and mitigation steps
Documentation describing the service and its dependencies, including the relevant on-call rotations to reach out if necessary

Having a team eager to jump in and help mitigate an outage quickly is priceless. Your team members understand some areas better than you. Knowing they have your back is reassuring.

2. Too many alerts and incidents

The second most common reason engineers fear their on-call is a never-ending litany of alerts, requests, and tasks. If you get a new alert when you barely finished acknowledging a previous one and are also expected to handle customer tickets and deal with requests from other teams, fretting your on-call is understandable. The exhaustion is usually exacerbated by the feeling of not doing a decent job. I was on a rotation like this once. After a while, I realized that everyone, not only me, was overwhelmed. Even though we toiled long hours, most alerts were ignored, customer tickets remained answered, and requests from other teams were only handled after they escalated them to the manager.

What to do about it?

There is no way a single person can fix a very heavy on-call by themselves. They won't have the time during their shift, and by the time the shift ends, they will be so fed up that they won't want to hear about anything on-call-related. There are, however, a few low-hanging fruits that can help improve the quality of the on-call quickly:

Delete alerts - find routinely ignored alerts and determine if they're useful. If they aren't - delete them.
Tune noisy but useful alerts - adjust thresholds and windows for flapping alerts, alerts that fire prematurely, and short-lived alerts.
Get a secondary on-call - a second person could help handle tasks the primary on-call does not have the capacity to deal with (e.g., customer tickets). This could be only temporary.

These ideas can alleviate on-call pain but are unlikely to fix a bad on-call for good. Improving a heavy on-call requires identifying and addressing problems at their source and demands effort from the entire team to maintain on-call quality. I wrote a post dedicated to this topic. Take a look.

3. Middle-of-the-night alerts

Many on-call rotations are 24/7. The on-call is responsible for dealing with incidents promptly, even if they happen in the middle of the night. Waking to an alert is not fun, and if it happens regularly, it is a valid reason to resent being on-call.

What to do about it?

While it may not be possible to avoid all middle-of-the-night alerts, there might be some actions you can take to reduce the disruption. A lot will depend on your specific situation, but here are some ideas:

Check your dashboards in the evening and address any issues that could raise an alert.
Increase alert thresholds outside working hours. If your traffic is cyclical - e.g., you have much lower traffic at night because most requests come from one timezone - you may be able to relax thresholds outside working hours. Even if an incident happens, its impact will be smaller. Also, alerts get much noisier if the traffic volume is low (e.g., if you get ten requests in an hour and one fails, you might get an alert due to a 10% error rate).
Disable alerts at night. Some outages won't cause any impact unless they last for a long time. For instance, our team was responsible for a service that would work fine even if one of its dependencies was down for a day. This 24-hour grace period allowed us to turn off alerts at night.

💙 If you liked this article...

I publish a weekly newsletter for software engineers who want to grow their careers. I share mistakes I’ve made and lessons I’ve learned over the past 20 years as a software engineer.

How to improve your coding skills (without spending a lot of time)

Pawel Kadluczka — Sat, 14 Sep 2024 04:43:26 +0000

Every developer wants to get better at coding. But they sometimes don't know how and feel stuck. The good news is that there are a few simple ways to improve coding skills that many developers neglect. Most can be done as part of a regular job, so they don't require additional time. And doing them consistently can make you a better coder.

Code reviews

Code reviews are the easiest way to learn from others. Unfortunately, many developers treat code reviews as a chore. They want to spend as little time as possible on them. This way, they lose great learning opportunities.

What makes code reviews special is their interactivity. They allow asking the author about their choices and discuss alternatives they considered. Other participants often leave interesting comments or propose novel alternative solutions.

Reading other developers' code

I learned a lot about programming by reading code written by other developers. I often want to know how a feature or library I use works. Usually, the fastest way to get this information is by inspecting its code.

For instance, TypeScript supported some features, e.g., async/await, before they were added to JavaScript. I was curious how it was possible. So, I wrote short TypeScript snippets and checked how they were transpiled.

Your company's codebase is another great resource to learn from. When I get stuck, I often search my company repository to check how other developers solved a similar problem. Our repo is big, so if I can't find anything useful, I am almost sure what I am trying to do is questionable.

I use the same strategy for my side projects but search GitHub.

If you find reading other developers' code challenging (I sure did), take a look at this post.

Debugging

Stepping through the code, analyzing the stack trace, and inspecting variables will allow you to understand important nuances and easy-to-miss details much deeper. I often fire a debugger if I can't answer all my questions after reading the code.

"Borrowing" code

Let's be honest. Not all code needs to be written from scratch. Sometimes, we just need a boilerplate. But sometimes, we don't know how to solve a problem. In these cases, copying and adapting code is often faster (and easier). It could be the code you wrote in the past or someone else's code, e.g., copied from StackOverflow (I have yet to find a software developer who never copied code from StackOverflow.) AI-powered programming tools are built on this idea. Tools like Github Copilot ask you to constantly vet and adapt the code they generate. Here is the thing, though. You'll learn nothing if you don't try to understand why the code you copied works or can't correctly adapt it.

Programming contests

Advent of Code taught me a lot. It is a light programming contest that takes place every December and consists of a series of small programming puzzles that can be solved in any programming language. I find it an excellent way to keep my coding skills sharp.

Solving Advent of Code problems is a good exercise, but examining other participants' solutions is where the real learnings are. And quite frankly, it can be a humbling experience. The different ways and techniques the participants use to solve the problems can be astonishing. I remember being proud of my ultra-short, 30-line-long solution, only to see someone else solve the same problem in the same programming language with just two lines of code because they used a clever idea.

💙 If you liked this article...

I publish a weekly newsletter for software engineers who want to grow their careers. I share mistakes I’ve made and lessons I’ve learned over the past 20 years as a software engineer.