DEV Community: Justin Fuller

Twilight of the Software Engineering Team

Justin Fuller — Mon, 05 Jan 2026 13:00:00 +0000

Does the advancement of AI-driven development call for a new type of team?

So far in my career, each of my teams has been staffed with some combination of: product manager, engineering manager, designer, data analyst, QA tester, and software engineer. Usually, a lot of software engineers.

Every one of those teams had two broad principles: first, move fast; second, make few mistakes.

Each team enacted these principles differently. For example, at a startup, we pursued speed with long hours and manual testing. At the banks, we stuck to tried-and-true tech stacks and rigorous testing protocols. Now, at The Times, my teams attempt to balance speed and correctness with developer tooling, such as robust CI pipelines.

Yet, regardless of each team's appetite for mistakes, they had one thing in common: the desire to move faster. There was always an ever-growing backlog of JIRA tickets. Those JIRA backlogs always grew faster than tickets closed. Ideas have always been cheaper to articulate than to implement. At least, until recently. Now, articulation and implementation are converging.

Even though the bottleneck was different for each team, it basically came down to one thing: engineering time. Occasionally, engineers would be stuck waiting on a product spec or a visual design, but that was rare. Even when this happened, the engineers typically had enough business context and design savvy to create a working MVP while waiting on the real specs and designs. In my career, I've only seen an engineer waiting around with nothing to do a few times. When all else fails, there's always some tech debt to clean up.

So, in my experience, the bottleneck has always been engineering. This is not an insult to any engineer's ability. These were teams of knowledgeable, efficient, highly capable engineers. Yet, we were always the bottleneck. This is because the process of engineering non-trivial software has historically taken a long time. Time spent hands-to-keyboard comprised only a small fraction of that time.

On any given project, engineers need to: understand the change, understand relevant parts of the code, determine how to implement the change, write the code, debug the code, test the code, optimize the code, deploy, and monitor. Code review and meetings occasionally (or often) disrupt these tasks.

Of course, some engineers and teams have fewer steps than these. Some have even more. But, in general, the shape of a project is roughly like that. It takes a lot of time.

Or, at least, it took a lot of time. Recently, our industry created (and parts of it have embraced) what can only be called dynamite for the old way of operating. This new technology can automate, to varying extents (based on both who you ask and where and how you use it), nearly all of the above tasks. And it's only getting better. New releases go out each month proclaiming that they are achieving even better results. More services are integrating with agentic coding assistants. The effectiveness and reach of the dynamite's blast only grows.

In my experience, these claims are valid. I have written about how I am producing all of my production, user-facing, business-critical software with agentic coding assistant tools like Claude Code. I am able to generate far, far more code than I could before. I now have a new problem: I can generate more code than either myself or my team can handle. It's now the humans who are slowing down the agents.

Software engineers producing code is no longer the bottleneck.

The very structure of our teams needs to change to accommodate this new world.

I believe this is the twilight of the software engineering team.

The Dawn of the Agentic Team

Now that we can produce 10x and potentially even 100x more code than we could before, we have a new problem. How can we manage such gargantuan quantities of code while maintaining quality? Every time I log in to LinkedIn, I see warnings about the perils of low-quality AI-generated code, "slop." It is clear many are running into the same problem.

One option is to arbitrarily slow down the teams to the pace they are currently capable of. This may be due to individuals rejecting the technological advancement, fear, or moral concerns. These fears and concerns are valid. They should be talked about and treated seriously. The moral concerns should be addressed, primarily politically but also by consumers voting with their wallets.

Right now, both politically and economically, the broad consensus appears to be that AI is the future (despite predictions of an imminent bubble pop). This applies to professional and personal settings. Claude Code is on my work computer, generating code with the blessing and funding of my employer. ChatGPT is on my phone, answering questions in my personal life. To be clear, this is not to say we should stop pursuing solutions to copyright, economic, and environmental issues. However, it does suggest those solutions will need to happen within the context of an AI-centered world. Abandonment seems to be a non-starter.

If ignoring AI won't work, the other option is to adapt. We must enable our teams to effectively manage increased output. Both options intend to maintain the level of quality. Only one attempts to maintain quality while increasing output.

What would this new team look like? For starters, it would need fewer people producing code. With Claude Code or similar tools, a single engineer can work on multiple projects at once. Personally, I've been able to effectively manage up to six simultaneous projects with agentic coding assistants. Beyond six, it can be quite difficult to manage. I believe with improved models and better tooling, our coding agents will require less babysitting. The number of concurrent projects per engineer will only increase.

So, we’ll need fewer Software Engineers focused on code output. That neither means we need fewer Software Engineers nor smaller teams. To the contrary, our ability to generate more code suggests we could use more people. We will produce more software than ever before. However, those same people will carry out vastly different tasks.

Before, the primary bottleneck was generating correct code. The new bottlenecks will be: identifying work, designing changes, creating technical specs, reviewing the sheer quantity of code produced, and managing the agent's supporting tools. These problems still require deep technical expertise — applied to different areas.

An experienced engineer on the team may now need to focus more on providing high-quality technical specs. These may take the form of JIRA tickets or similar. Those specs will be an important part of the agent’s prompt.

Another engineer (or multiple) may need to focus on reviewing the enormous volume of code. Agents could use their feedback to iterate on a PR. These engineers would not generate code but spend the bulk of their time reading, validating, and articulating feedback for the agents to act on.

Other engineers may need to focus on building tools for agents, much like some engineers today may focus on building pipelines and developer tooling. These engineers may connect the agent to other tools like JIRA, GitHub, Slack, PagerDuty, Google Docs/Sheets, and Datadog. The possibilities are endless. The goal of this engineer would be to continually streamline the use of agentic coding assistants.

This is still a highly skilled, technical team. However, the shift in focus would reflect moving bottlenecks caused by evolving technology.

Caveats

In reality, AI may produce teams that are very different from what I’ve described above. I do not feel confident about any particular team shape. Instead, I feel confident about two things.

First, I believe teams that do not adjust to this new world will continually feel frustrated. They will experience the volume of output created by agentic tools as a detriment, and rightfully so. The current shape of Software Engineering teams is not equipped to handle this volume. Imagine a hose designed to output 20 liters of water per minute. You’re going to have a bad time if you suddenly try to 10x that volume.

I am not saying those teams will not succeed. Just like there are successful teams still working with waterfall instead of agile, there will still be successful Software Engineering teams in the AI era. I am specifically warning against the mixing of high-output agentic coding assistants with a human-coder-centered team structure.

Second, I believe the evolution will be messy. This is a rapidly evolving technology. Teams should continually experiment with new structures. Some will fail. Some will work for a time, then stop working when the technology advances further. I believe this will be an uncomfortable but healthy part of the industry’s evolution. Engineering leaders should embrace this discomfort and adopt an experimentation mindset with their teams. They should try five different structures in five different teams, then adopt whatever works best, wherever it works best.

Some teams may need to stack heavily on code review. Others may need to focus on automation. Still others, testing. The bottleneck has shifted, but we do not yet understand how to best adapt to it. My intention with the description above is to articulate just one possibility and to encourage the industry to imagine what teams could be like in this new world.

This post originally appeared on justindfuller.com. You can follow me on LinkedIn and GitHub.

Keeping One Tab Open

Justin Fuller — Sun, 18 Dec 2022 01:14:00 +0000

This post originally appear at justindfuller.com

Recently, someone found the maximum number of tabs that Microsoft Edge can handle. The number is sixteen hundred. One tab over this and it crashes.

Not too long ago, I felt like my brain had sixteen hundred tabs open at all times. Even while I focused on one task, I would keep several tabs open in the background, mostly to remember to do things after the current task.

I’d try to remember:

What I planned to do next
The email or slack message I need to follow up on
Questions I need to ask my team
Issues I need to address
And so much more

Even when I focused deeply on a task, I still had to dedicate a large portion of my processing power to all these other “open tabs” that I needed to hold on to.

It’s not just a metaphor, though. This was reflected in the literal open browser tabs on my computer.

At any given time, I would have open:

The current page I’m working on, perhaps adjusting a component or adding a feature
MDN or StackOverflow to answer some question I have, related to the current task
Maybe a few more MDN and StackOverflow tabs that might be useful
Gmail, google calendar, jira, trello
Some blog posts I want to read later
Random tabs I opened and forgot about but I’m not sure I’m ready to close them yet
And probably a lot more too

This trend of open tabs and “open tabs” continued until I discovered the book, Getting Things Done: The Art of Stress-Free Productivity.

Write It Down

Most people laugh or don’t believe me when I try to explain how helpful this method is, because it sounds too simple.

If you have something you need to remember: write it down.

Once you write it down in a place that you can easily come back to later, you can finally close that open tab. Once you write down everything that you need to remember, you can close all of your open tabs and start to truly focus on one thing at a time.

For this to work, you need to write it down in a place that…

Is consistent across all the things you think about
You can trust to remind you
Allows you to prioritize and reprioritize tasks

Trello

For the rest of this post, I will show you how I use Trello to keep one tab open. However, Trello is just one of many tools that you can use. It probably will not be right for everyone. I hope what you’ll see is the strategy I use, rather than the specific tool I use to accomplish it.

Step 1: Record All Inputs

Throughout your day you’ll receive countless inputs.

Emails
Old-school snail-mail
Slack messages
Texts
Calendar Invites
Your spouse asking you to do something
You’ll notice an issue with your house or work
You’ll remember something you forgot to take care of

Typically, you have a few options to handle these inputs:

Completely ignore it
Try to remember it for later
Drop what you’re doing to handle it immediately

But there is another way: record the input, along with any relevant information such as dates or links, so that you can come back to it later.

For me, this looks like dropping a new card in my TODO column in Trello. When I notice a slack message or email that requires follow-up, I drop a card in the column.

Step 2: Add Detail

The cards alone are useful as a prioritization and reminder tool. However, the real benefit materializes only when you are able to decrease your mental load and “close some tabs”.

That’s where adding detail comes in.

When I create a card, I add:

Any relevant links to get me back to the slack message, email, or related information
A brief description containing whatever thoughts I have on the task.
- “Make sure to talk to XYZ person about this before responding”
- “See if this has already been implemented in another system”
A due date if there’s a deadline or even ideal time to have this task complete

Details are crucial to staying focused. With only the card you have a reminder of what you need to do, but not how to do it. So you may find that the task itself is off your mind, but you’re still attempting to hold on to all the extra things required to complete the task.

Step 3: Focus

This is the hardest part. You now have the tools to focus on one task at a time. But it still takes conscious effort to do so.

In Trello, this looks like literally one card in the “Doing” tab at all times. There should always be a card in that tab unless you are actively choosing to be “off” or “wasting time” (which is a totally valid and normal thing to do, by the way, life isn’t only about productivity).

Switching Tasks

It’s pretty common to start a task, only to be interrupted and forced to switch to another task. This is notoriously difficult for “knowledge workers” like Software Engineers. However, your cards can help you context switch easily.

When switching tasks, you need to be able to completely turn off the current task and focus on the new one.

Write down any new details you’ll need to remember when coming back.
Add relevant links, such as the path to the file you were working in or the StackOverflow answer you were reading.
Move the task back to “TODO”
If the new task will take more than 5 minutes: Create a card for it and move it to “Doing”.

Finishing Tasks

Whenever you finish a task, move it to the “Done” column.

At the end of each week, I review the completed tasks to see if I’m satisfied with my work.

I like to ask myself:

Did I spend my time wisely this week?
Did I get stuck on a certain task for too long?
What were the affects or results of the tasks I completed?

If I’m not satisfied, I try to understand why, and prepare to spend my time differently in the next week.

Cleaning Up

The more you do this, the more you will accumulate cards. At the end of each week, I move my “Done” column to a different board. I don’t delete the cards, because I occasionally like to browse back through them to do a longer retro on my work.

I also set up an automation to remove cards that are older than thirty days. Again, it does not delete the cards, it moves them to an “Archive” board. I do this because I have a lot of ideas, but often find that they are unimportant to me a month later. I think it’s healthy to let go of ideas once I no longer feel they are relevant. I don’t need to power through unimportant tasks, just because I incorrectly thought they were important in the past.

Putting it All Together

Write down every input
Record enough details that you can truly get it off your mind
Focus on one task at a time
Do a weekly retro on your completed tasks
Regularly prune your tasks to focus on only what is currently relevant

Why do we fall into the rewrite trap?

Justin Fuller — Tue, 21 Jan 2020 12:20:25 +0000

This post originally appeared on JustinDFuller.com.

One of my favorite reads is Joel Spolsky's Things You Should Never Do. He wrote this post almost twenty years ago, outlining the downfall of Netscape and others because they spent years rewriting working code. His solution is, unsurprisingly, to refactor. About a year before Joel wrote Things You Should Never Do, Martin Fowler published his popular book, Refactoring: Improving the Design of Existing Code.

So, my question is, if we as a community figured out — twenty years ago — that we should stop rewriting programs, why is it still commonly done today?

My History

To figure out why we keep falling into the rewrite trap, I want to first step back through a few points in my history.

Cold–what?

My first job was with a fun, scrappy startup called SignUpGenius. Most of SignUpGenius's code had been written before I was even thinking of becoming a web developer, on a platform called Coldfusion (you've probably never heard of it), with a sprinkling of Angular.js. We had no microservices, no Node.js, no React, Go, Docker, or Kubernetes. Our production deployment consisted of arriving early in the morning to manually copy over files and debug issues (I told you we were scrappy).

Yes, this situation is not ideal and, no, I am not advocating for anyone to manually FTP files for their production deploys. However, one thing that cannot be denied is that we were very productive. Granted, the Coldfusion code was old, but we were constantly improving it. There was never talk of a rewrite; instead, the code was improved, little by little, each time we worked on adding a feature or fixing a bug.

The results were clear: in my time at SignUpGenius I was able to see the company gain market share; with increased traffic, revenue, and even a buyout of a competitor.

The code wasn't perfect — far from it, as any codebase would be after ten years — but if we had stopped to rewrite the whole thing we would have been dead in the water and the company would have suffered. Just like in Joel's post, where Netscape took years to rewrite its already working code, rewriting code because Coldfusion was "dead" or "inferior" would have served little business purpose.

I understand there are plenty of counter-arguments to be made here. First and foremost, it gets harder every day to find a Coldfusion developer. Those who know the platform are most likely looking to give a facelift to their skillset, rather than prolonging the inevitable drying-up of the proverbial job pool. More, it may be harder to implement functionality in Coldfusion, particularly the kind we were dealing with, than, say, in .Net, Ruby on Rails, or even Go.

Even so, there are other, better ways to move toward new technology without rewriting an entire codebase.

Taking advantage of new technologies

If I think SignUpGenius was doing the right thing by not rewriting, how, then, how does one take advantage of new technologies?

This brings me to the next chapter of my professional career, when I found myself at FIS Global (the biggest company you've never heard of), working on user experiences with the wealth and retirement division.

Our team found a very specific problem: the websites for 401k plan participants and sponsors were horrendous. Not just in how they looked (but they did look pretty terrible) but also in how participants could interact with them.

The company had built heavily around .NET, which, while it has its merits, did not provide the type of experience that was best for this environment.

To understand why, you must first understand that interactions with your 401k plan are not like your interactions with, say, Amazon. On Amazon, you are probably going to a specific shopping page that you want to load very quickly. You're also probably not very likely to use more than a couple of pages of the website (possibly a search page, a few products, and the checkout). The pages are not necessarily related to each other. There's no reason to load them all up at once, in fact, there's plenty of reasons not to.

However, your retirement website is very different. When you go to it you are not trying to load it as quickly as possible to make an impulsive purchase of some ETFs — you're probably doing your yearly check-in or enrolling for the first time. You'll probably check your balance, your contributions, the performance of your holdings, and maybe even download a few tax documents.

We found that users were coming less often, while spending far more time on the website, using many parts of it — the perfect use-case for a single-page web application. The initial load time was a low priority while seamless transitions between pages were of great value.

So, what did we do? Did we rewrite the whole application? No.

Existing applications' UI code was refactored, reorganized, and maybe even rewrapped with a framework like Angular. New applications were kept separate and had new code written with the new paradigm.

This allowed for a quick, gradual, and seamless transition from old technology to new, with very little loss of all those years worth of things like tests and bug fixes.

Contempt Culture

Unfortunately, some developers and institutions bank hard on convincing others that a rewrite is the best use of their money. To do this they rely on convincing others of the inferiority of older, battle-tested languages like Java and PHP to newly popular technologies like Node.js or Go.

They don't just stop at "Go is better than Java for this problem" — no, they say Java is always bad, always the wrong choice, and our existing Java projects are terrible messes that need to be rewritten. Maybe, even, the experienced Java developers that we have aren't as good as we once thought.

This is contempt culture and it's prevalent in some software development communities. I know this because I was one of the developers who called old languages terrible, clunky, slow while claiming the superiority of my language of choice.

That is until I started learning about and using some of those older languages.

When I read through Robert Martin's Clean Code Series I was struck by how easy it was to understand the Java and C++ code that he wrote. I realized that the Java and C++ he was writing was far easier to understand than the JavaScript and Go that I had been writing for a few years. The same thing happened when I read through Martin Fowler's Refactoring — he transformed hard to understand, hard to change Java into code that is clear and easy to change to meet new requirements.

After starting a new job at The New York Times — which has amazing code quality, and not just for an organization that is over 150 years old — I found that the company still has some legacy Java and PHP services. I was scared when I heard that I needed to research how to interact with these APIs — until I saw the code. It turns out that these services are so well-factored, with such well-established patterns, that they were incredibly easy to understand. Less than two months into the job and I was able to traverse several of these codebases to find exactly what I needed.

So much for contempt culture, these older languages can be great.

The rewrite

Back to rewrites. I mentioned earlier that some developers rely on convincing their managers that a rewrite is necessary; in some cases, even multiple-year, large team efforts to completely rewrite existing codebases. I've even seen a case where there were multiple rewrites in a row. Did no one stop to think, "if I didn't get it right the first few times, why would I get it right this time?"

No, I don't finally have the right programming language. Maybe I finally have the right team, but probably not.

I think Joel is exactly right with his fundamental, cardinal rule of programming:

It's harder to read code than to write it.

I recently experienced this first-hand. In the codebase I'm currently working with, I encountered an error handling pattern that I didn't like; it seemed cumbersome and convoluted.

My initial thought was to ignore this pattern, rewrite it, maybe even replace it with a better solution. That was until I found that the original author put together an entire talk about this error handling pattern. It turns out that, while it may be a little confusing, it gives incredible transparency when you need to debug with application logs.

The real problem is that I hadn't taken the time to understand how the error pattern worked. I was confused, which led me to dismiss it, rather than to improve it.

👋 Want to learn more about programming? Subscribe to my newsletter to get an update, once-per-month, about what I'm writing about.

When to Refactor

So, if rewrites are so bad, how can you avoid them?

Here's a rule of thumb: If your reason for rewriting the code is that you don't understand it, you should not rewrite it. Instead, you should spend the time trying to understand the code. Once you understand it, refactor the code to make it easier for the next person to understand.

Instead of rewriting the error handling pattern I will probably try to improve a few method names. This brings me back to Martin Fowler's Refactoring. Most of the time, if you feel that a rewrite is needed, you can probably just refactor a few pieces of the program and you'll be in great shape.

If your reason for rewriting the code is that you don't understand it, you should not rewrite it.

Sure, refactoring may have more pieces now than it did when he wrote it — moving an endpoint to another service may be a modern extension to moving a method to another class — but the base concepts are still the same.

More importantly, the benefits are still the same. You will retain all the bug fixes, handled edge-cases, and sparsely-documented features that you don't even know about. They'll still be there when you're done refactoring.

If you rewrite, you'll likely lose much of that. It's an idealistic view of ourselves to think that we understand any large codebase, probably rewritten by several or dozens of developers over months or years, well enough to cover all of these cases in a rewrite. Much of it will be lost.

Hurdles to Refactoring

I used to have a very abstract understanding of refactoring. My manager would ask if I was done with my current task and I would say, yes, but I need to refactor it before it's ready.

What did I mean by that? In the beginning, I had some vague notion of making the code "prettier" or "easier to understand". They're not bad reasons; if I just wrote code that I can't understand now, I certainly won't be able to when I come back a few months later.

The problem with this approach is that a vague understanding of refactoring comes with a vague understanding of the costs and benefits. Will the next developer agree that the code is easier to understand?

Imagine that you were asked to clean up after a party. If you have never cleaned up before or if it was a big party, it could seem like a daunting task. How do you clean up? Should you remodel the bathroom or should you throw away all the cups and plates? Should you mow the yard, or make the living room usable again by putting all the chairs back at the table?

Books like Refactoring help by giving a clear picture of when and how to refactor. This greatly reduces the cost of refactoring because the possibilities are no longer limitless.

For example, here are a few concrete refactoring reasons and solutions:

Reason 1: The code is hard to understand. I don't know where to make my change.

Solution 1: Read through the code; add tests where there are none; then, once you understand their true purpose and have added tests, improve variable or function names.

Reason 2: Making this change will touch many parts of the codebase, otherwise known as shotgun surgery, I'm worried something will break.

Solution 2: Rearrange the codebase so that your change will only have to modify one or a few pieces of the code.

You can use code smells and their specific solutions to greatly reduce the abstract-ness of refactoring. Refactoring can become a quick, concrete tool that you use to accomplish specific purposes.

Backward Refactoring

I used to think refactoring was done after I finished coding. I finished my work, I made some concessions in the name of "Make it work, make it right, make it fast", now I need to clean up.

However, refactoring can be a better tool before you begin making any changes as a way to make your change easier. As Kent Beck said "Make the change easy (warning: this may be hard), then make the easy change."

// Detect dark theme var iframe = document.getElementById('tweet-250733358307500032-216'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=250733358307500032&theme=dark" }

Think about when you refactor after making your changes. What are you doing? You're guessing what will be needed by the next person or the next change to the code. You're guessing what will be misunderstood or unclear to the next developer. You might be right, but you also might be wasting everyone's time.

When you refactor before making a change you have a clearer picture of what is misunderstood, what change needs to be done, and why it's difficult to accomplish.

For this reason, I suggest another rule of thumb: Prefer refactoring before you make a change, rather than after.

When to rewrite

As always, things aren't perfectly black-and-white. There are a few times when a rewrite might be necessary.

The golden opportunity to rewrite less-than-stellar code is when the business wants to re-think how the product works. If the business is unhappy with the product and the developers are unhappy with the code, this may be the perfect opportunity to rewrite. Rewrite for better functionality, rather than better code. However, unless you're in a fast-moving startup, this opportunity may not come very often as most companies that I've worked with prefer incremental change.

This is one reason that, as Sam Newnan writes in Building Microservices, some people suggest to keep microservices small enough to rewrite in 2 weeks or less. There's no huge loss if a total overhaul is needed. You could rewrite it during a slow season.

There's another opportunity, which (I hope) is even rarer than the last case. Sometimes a development team has written code that is so bad, so convoluted, that even they can't understand it enough to make a single change without creating a slew of bugs. If this is the case, a rewrite may be unavoidable.

Rewrite for better functionality, rather than better code.

However, let me caution you even in this case. If the composition of the team has not drastically changed; if there is not a clear definition of the cause and solution to the last iteration's problems; or there is not new leadership that provides clearer vision, principles, and guidance — you will likely repeat the same mistakes all over again. If the people haven't changed, the circumstances haven't changed, and the practices haven't changed, what will be different this time?

Key Takeaways

Yes, all of this was just to say that you should prefer refactoring over rewriting. If you couldn't tell by all the times I linked to it, I highly recommend Martin Fowler's Refactoring to learn more about the subject.

Here are the key takeaways:

Prefer refactoring over rewriting.
If your reason for rewriting the code is that you don't understand it, you should not rewrite it.
Prefer refactoring before you make a change, rather than after.
Rewrite for better functionality, rather than better code.

Hi, I’m Justin Fuller. I’m so glad you read my post! I need to let you know that everything I’ve written here is my own opinion and is not intended to represent my employer. All code samples are my own.

I’d also love to hear from you, please feel free to follow me on Github or Twitter, or subscribe to my newsletter to get an update, once-per-month, about what I'm writing about. Thanks again for reading!

Go Things I Love: Channels and Goroutines

Justin Fuller — Mon, 06 Jan 2020 13:19:00 +0000

This post originally appeared at JustinDFuller.com.

This series, Go Things I Love, is my attempt to show the parts of Go that I like the best, as well as why I love working with it at The New York Times.

In my last post Go Things I Love: Methods On Any Type, I demonstrated a feature of Go that makes it easy to build Object-Oriented software.

This post, Channels and Goroutines, will demonstrate a few neat concurrency patterns in Go.

First: to get the most out of this post you should familiarize yourself with the fundamentals of Go concurrency. A great place to do that is in the Go tour. These patterns rely on goroutines and channels to accomplish their elegance.

First Class

Concurrency, in some form, is one of the most important building blocks of performant software. That's why it's important to pick a programming language with first-class concurrency support. Because Go, in my estimation, provides one of the most delightful ways to achieve concurrency, I believe it is a solid choice for any project that involves concurrency.

To be first-class is to have full support and consideration in all things. That means, to be first-class, concurrency must be a part of the Go language itself. It cannot be a library bolted on the side.

A few type declarations will begin to show how concurrency is built into the language.

type (
    WriteOnly(chan<- int)
    ReadOnly(<-chan int)
    ReadAndWrite(chan int)
)

Notice the chan keyword in the function argument definitions. A chan is a channel.

Next comes the arrow <- that shows which way the data flow to or from the channel. The WriteOnly function receives a channel that can only be written to. The ReadOnly function receives a channel that can only be read from.

Being able to declare the flow of the data to a channel is an important way in which channels are first-class members of the Go programming language. Channel flow is important because it's how goroutines communicate.

It's directly related to this phrase you might have seen before:

Do not communicate by sharing memory; instead, share memory by communicating.

The phrase, "share memory by communicating", means goroutines should communicate changes through channels; they provide a safer, idiomatic way to share memory.

Communicating by sharing memory (👎)

Here's an example of Go code that communicates by sharing memory.

func IntAppender() {
    var ints []int
    var wg sync.WaitGroup

    for i := 0; i < 10; i++ {
        wg.Add(1)

        go func(i int) {
            defer wg.Done()
            ints = append(ints, i)
        }(i)
    }

    wg.Wait()
}

IntAppender creates a goroutine for each integer that is appended to the array. Even though it's a little too trivial to be realistic, it still serves an important demonstrative purpose.

In IntAppender each goroutine shares the same memory—the ints array—which it appends integers to.

This code communicates by sharing memory. Yes, it works—but it's not idiomatic Go. More importantly, it's not the safest way to write this program.

It's not very safe because there are 11 goroutines (one running the main function and ten more spawned by the loop) with access to the ints slice. What happens when the codebase grows?

This pattern provides no guarantee that the program will behave as expected; anything can happen when memory is shared broadly.

Share memory by communicating (👍)

The first sign that this example is not following "share memory by communicating" is the use of sync.WaitGroup. Even though I consider WaitGroups to be a code smell, I'm not ready to claim they are always bad. Either way, code is usually safer with a channel.

Let's convert the bad example to idiomatic Go by replacing the WaitGroup with a channel.

// WriteOnly serves the purpose of demonstrating
// a method that writes to a write-only channel.
func WriteOnly(channel chan<-int, order int) {
    channel <- order
}

func main() {
    var ints []int
    channel := make(chan int, 10)

    for i := 0; i < 10; i++ {
        go WriteOnly(channel, i)
    }

    for i := range channel {
        ints = append(ints, i)

        if len(ints) == 10 {
            break
        }
    }

    fmt.Printf("Ints %v", ints)
}

See this example in the Go playground.

Now, only one goroutine can modify the ints slice while the rest communicate through a channel. They're sharing memory by communicating through a channel instead of modifying shared memory.

The example here shows two important ways that concurrency (goroutines and channels) are first-class citizens of the Go programming language. First, we used a write-only channel argument. This guaranteed that the method won't accidentally read from the channel, unexpectedly altering the functionality. Second, we see that the for range loop works on channels.

These are just a few ways that Go makes concurrency a first-class citizen. Next, let's see what we can accomplish with goroutines and channels.

Timeout

To demonstrate a timeout, we will construct a simple news UI backend that fetches results from three New York Times endpoints. Even though the NYT endpoints respond very quickly, this won't quite meet our standards. Our program must always respond within 80 milliseconds. Because of this restriction, we're only going to use NYT endpoint responses that come fast enough.

Here are the URLs that the program will fetch from:

var urls = [...]string{
    "https://api.nytimes.com/svc/topstories/v2/home.json",
    "https://api.nytimes.com/svc/mostpopular/v2/viewed/1.json",
    "https://api.nytimes.com/svc/books/v3/lists/current/hardcover-fiction.json",
}

The URLs have been declared as an array of strings, which will allow them to be iterated.

Another neat feature of Go is how you can declare const blocks. Like this:

const (
    urlTopStories              = "https://api.nytimes.com/svc/topstories/v2/home.json"
    urlMostPopular             = "https://api.nytimes.com/svc/mostpopular/v2/viewed/1.json"
    urlHardcoverFictionReviews = "https://api.nytimes.com/svc/books/v3/lists/current/hardcover-fiction.json"
)

Now, the urls array can be more expressive by using the const declarations.

var urls = [...]string{
    urlTopStories,
    urlMostPopular,
    urlHardcoverFictionReviews,
}

The URLs are for top stories, most popular stories, and the current hardcover fiction reviews.

Instead of a real http.Get I will substitute a fake fetch function. This will provide a clearer demonstration of the timeout.

func fetch(url string, channel chan<- string) {
    source := rand.NewSource(time.Now().UnixNano())
    random := rand.New(source)
    duration := time.Duration(random.Intn(150)) * time.Millisecond
    time.Sleep(duration)
    channel <- url
}

This is a common pattern in Go demonstration code—generate a random number, sleep the goroutine for the randomly generated duration, then do some work. To fully understand why this code is being used to demonstrate a fake http.Get, the next sections will step through each line, explaining what it does.

Deterministic Randomness (See: oxymorons)

In Go, the random number generator is, by default, deterministic.

In mathematics, computer science and physics, a deterministic system is a system in which no randomness is involved in the development of future states of the system. - The Encyclopedia of Science

This means that we have to seed the randomizer with something that changes; if not, the randomizer will always produce the same value. So we create a source, typically based on the current time.

source := rand.NewSource(time.Now().UnixNano())

After the source is created, it can be used to create a random number generator. We must create the source and random generator each time. Otherwise, it will continue to return the same number.

random := rand.New(source)

Once the generator is created, it can be used to create a random number between 0 and 150. That random number is converted to a time.Duration type, then multiplied to become milliseconds.

duration := time.Duration(random.Intn(150)) * time.Millisecond

One further note about the randomness is needed. It will always return the same value in the go playground because the go playground always starts running with the same timestamp. So, if you plug this into the playground, you'll always receive the same result. If you want to see the timeout in action, just replace 150 with some number below 80.

Another send-only channel

At the very bottom of fetch are the two lines that we care about.

time.Sleep(duration)
channel <- url

The first line tells the goroutine to sleep for the specified duration. This will make some responses take too long for the given URL, later causing the API to respond without the results of that URL.

Finally, the URL is sent to the channel. In a real fetch it would be expected that the actual response is sent to the channel. For our purposes, it's just the URL.

A read-only channel

Since the fetch function funnels results in the channel, it makes sense to have a corresponding function funnel results from the channel into a slice of strings.

Take a look at the function. Next, we'll break it down line-by-line.

func stringSliceFromChannel(maxLength int, input <-chan string) []string {
    var results []string
    timeout := time.After(time.Duration(80) * time.Millisecond)

    for {
        select {
        case str := <-input:
            results = append(results, str)

            if len(results) == maxLength {
                fmt.Println("Got all results")
                return results
            }
        case <-timeout:
            fmt.Println("Timeout!")
            return results
        }
    }
}

First, look at the function argument declaration.

func stringSliceFromChannel(maxLength int, input <-chan string) []string {

The stringSliceFromChannel function declares that it will accept a read-only channel, channel <-chan string. This indicates that the function will convert the channel's inputs into a different type of output—a slice of strings, or []string.

Even though it's valid to declare a function argument with, channel chan string, opting for the arrow <- operator makes the function's intent clearer. This can be particularly helpful in a long function.

Next, the timeout is created.

timeout := time.After(time.Duration(80) * time.Millisecond)

The function time.After returns a channel. After the given time.Duration it will write to the channel (what it writes doesn't matter).

Moving on, the timeout and input channels are used together in a for select loop.

The for loop with no other arguments will loop forever until stopped by a break or return.

The select acts as a switch statement for channels. The first case block to have a channel ready will execute.

By combining the for and select, this block of code will run until the desired number of results is retrieved or until the timeout happens.

Take a look at the case block for the input channel.

case str := <-input:
    results = append(results, str)

    if len(results) == maxLength {
        fmt.Println("Got all results")
        return results
    }

The output of the channel is assigned to a variable, str. Next, str is appended to the results array. The results array is returned if it is the desired length.

Now, look at the case block for the timeout channel.

case <-timeout:
    fmt.Println("Timeout!")
    return results

Whatever results are available, even if there are none, will be returned when the timeout happens.

👋 Want to learn more about Go? Subscribe to my newsletter to get an update, once-per-month, about what I'm writing about.

The Main Function

Now there is both a channel writer and a channel reader. Let's see how to put it all together in the main function.

func main() {
    channel := make(chan string)
    for _, url := range urls {
        go fetch(url, channel)
    }

    results := stringSliceFromChannel(len(urls), channel)

    fmt.Printf("Results: %v\n", results)
}

First, a channel is created to collect the fetch results, channel := make(chan string).

Next, the urls are looped over, creating a goroutine to fetch each url.

for _, url := range urls {
    go fetch(url, channel)
}

This allows the fetching to happen concurrently.

After the fetches have been kicked off, stringSliceFromChannel will block until either the results are in or the timeout occurs.

results := stringSliceFromChannel(len(urls), channel)

Finally, we can print the results to see which URLs are returned. If you run this code in the Go Playground, remember to change the timeout number since the random number generator will always return the same results.

Final Thoughts

Here's the cool thing. We started out talking about how Go has first-class concurrency support with goroutines and channels. Then we saw how easy it is to implement a complex concurrent pattern, a timeout, with a single channel and a few goroutines. Over my next few posts, I hope to show that this was only scratching the surface of what one can do with concurrency in Go. I hope you'll check back in. (Better yet, subscribe to my newsletter to be updated each month about my new posts)

Finally, even though this is a neat concurrency pattern, it's unrealistic. As an exercise you could open the Go Playground to see if you can implement these scenarios:

The results should be returned as a JSON object. Maybe we could use a struct instead of an array of URLs?
A blank page is useless, the code should at least wait until there is one result to display.
The context type is often used with http handlers. Can you replace the time.After with an expiring context?

Hi, I’m Justin Fuller. Thanks for reading my post. Before you go, I need to let you know that everything I’ve written here is my own opinion and is not intended to represent my employer. All code samples are my own.

I’d also love to hear from you, please feel free to follow me on Github
or Twitter. Thanks again for reading!

Go Things I Love: Methods On Any Type

Justin Fuller — Sat, 14 Dec 2019 20:41:59 +0000

This post originally appeared on JustinDFuller.com.

Now that I am working with Go as my primary language at The New York Times, I want to explore some of my favorite features of the language. I don't intend this to reveal previously unknown features or best practices; I just want to share some of the reasons that I enjoy working with the language.

Methods

Typically a method is defined as a function that belongs to an object or class. This means you wouldn't necessarily add a method to a string, boolean, or number.

This limitation might lead a developer to produce code that looks like this:

const LastOldSchoolID = 23959

if id < LastOldSchoolID {
  // allow access to the old school version of the game
}

// and elsewhere

if id < LastOldSchoolID {
  // allow access to the old school version of the forum
  // or some other custom logic for old school players
}

In this silly example, there is a game that recently released a new version. Players who made an account before the release are still allowed to play the old game and use the old forum. To do this, the developers went around the codebase and added a check: Is the user's ID lower than the last old school player's ID? If so, they can access the old game.

Encapsulation

Unfortunately, and right away, the community notices a bug. Specifically, a single user notices this bug. The last old school player is locked out of the game! The developers used the check id < LastOldSchoolID everywhere. So it works for all but the very last player.

At this point, the developers are forced to search for every instance of this check (fortunately there aren't that many, only a dozen or so) and they replace the logic with id <= LastOldSchoolID. Everything is working perfectly again.

Except it's not. The developer who coded logic for the old school login page included this snippet:

if id >= LastOldSchoolId {
  return httperrors.Unauthorized("We're sorry but you don't have access to the old school game.")
}

So, unfortunately, the bug still exists.

This could have all been prevented by a little encapsulation. The developers shouldn't have repeated the logic—all over the codebase—to determine if an ID is valid for old school. The logic should live in only one place.

Thankfully, Go provides exactly what is needed.

Using a custom type

The User's ID will be implemented as a custom type. The type can be used independently or as part of a user struct. Most importantly, it can have custom methods.

Here's how it works:

const LastOldSchoolID = 23959

type UserID int

func (id UserID) IsOldSchool() bool {
  return id <= LastOldSchoolID // The last oldschool player
}

func (id UserID) IsNotOldSchool() bool {
  return !id.IsOldSchool()
}

Notice the custom methods added directly to the UserID type, which is an int. The code around the app can now be rewritten.

if id.IsOldSchool() {
  // allow access to the old school version of the game
}

// elsewhere

if id.IsNotOldSchool() {
  return httperrors.Unauthorized("Please join us playing the new version at game.com/v2.")
}

If only this had been done in the first place, a lot of pain and suffering would have been saved for that poor user who wasn't able to access the game for a few days. The developers could have made the fix at a single location in the code and the fix would have been applied everywhere.

More encapsulation, delegation.

Some might point out that the code is still revealing too many details about the user and the old school logic.

Why should the rest of the code need to know that old school access is determined by the ID? What if the business decides to change the rules to instead use the creation date? What if the developers decide to make an IsOldSchool property on the user?

These are all valid points. First, I ask you to remember that this is simply an example to show a great feature within Go.

Next, I'd like to point out two things.

Adding methods to a non-struct type (like an int or string) might be a code smell that you are unnecessarily leaking implementation details or other logic that should be private.
If the details need to be hidden, methods can still be added to the custom type, while at the same time hiding where that logic comes from.

Allow me to demonstrate.

type User struct {
  UserID
}

func (u User) IsOldSchool() bool {
  return u.UserID.IsOldSchool()
}

func (u User) IsNotOldSchool() bool {
  return u.UserID.IsNotOldSchool()
}

Delegation is used inside the User struct. The details of how the ID calculates old school is even hidden to User.

Now the implementation details (that old school access is based on the ID) can be hidden from the rest of the code.

if user.IsOldSchool() {
  // allow access to the old school version of the game
}

// elsewhere

if user.IsNotOldSchool() {
  return httperrors.Unauthorized("Please join us playing the new version at game.com/v2.")
}

Again, internally, the User struct will take advantage of the methods on the ID, further encapsulating the logic.

You can play around with these examples on the Go Playground.

I’d also love to hear from you, please feel free to follow me on Github or Twitter. Thanks again for reading!

Keep a git repository of all your practice code

Justin Fuller — Sun, 13 Oct 2019 18:59:22 +0000

This post originally appeared on JustinDFuller.com.

Are you struggling to find projects to showcase to potential employers? Have you been practicing LeetCode as you prepare to interview for a Software Development job? Do you occasionally practice a Kata on Codewars to keep your skills from getting rusty?

If you answered yes to any of these questions, you should consider saving every practice problem you do in a public git repo, like Github.

I try to keep up with programming trends, and one question that I see come up a lot, particularly on reddit, is "What project should I do?"

Projects are a great way to learn, but most people don't have time to work on big projects in their free time. Another option is to work on algorithmic problems. You've got an almost 100% chance of running into at least one algorithm question if you are interviewing, plus they are a good way to learn a new programming language.

If you have been a developer for years, practicing algorithm problems is a great way to keep your skills sharp—particularly on a groggy Monday morning when you are having trouble getting started, or on a Friday afternoon when you are feeling unmotivated. A quick practice problem can get your brain moving.

My knowledge repo

I recently started learning C++. To keep track of what I am learning I created a practice problem repo of my own that I hope to see grow over time.

What do I plan to do with it?

Code review myself.
- From time to time I will come back to the code and review it. I'll ask myself, "based on what I have learned, is this still the best way to implement this?" If the answer is no, I will create a new one and document what I misunderstood when I originally wrote the code.
Use it as a knowledge repository.
- I may have a similar problem pop up in a real application in the future. Now I have a knowledge repo that demonstrates how I solved it the first time.

Your knowledge repo

What can you use it for?

Learning a new programming language.
- Practice each new concept you learn and save it for review later; make sure you don't just copy and paste examples, though!
Tracking your improvement over time.
- Occasionally review your old code. What have you learned since you wrote it?
A personal knowledge repository.
- When you get a tough problem at work, maybe you've done something similar before—now you have a way to remember.
A showcase for potential employers.
- New college graduate or long-time developer, there's no better way to show your skill than to have a potential employer see your code.

I’d also love to hear from you, please feel free to follow me on Github or Twitter. Thanks again for reading!

Simply JavaScript: a straightforward intro to Mocking, Stubbing, and Interfaces

Justin Fuller — Wed, 04 Sep 2019 01:13:29 +0000

This post originally appeared on JustinDFuller.com.

I like to think that I’m a simple guy, I like simple things. So whenever I sense complexity, my first reaction is to wonder if I can make things easier.

Before I transitioned to software development, I spent time as a sound engineer. I was recording bands and mixing live shows. I was even recording and mixing live shows for broadcast. During that time I talked with too many people who would always attempt to solve problems by purchasing some expensive, more complex equipment. Sadly the return on investment never seemed to be all it promised.

Instead of buying into the “more expensive, more complex is better” philosophy, I spent every evening learning the basics. I focused on fundamental skills. I learned how to use an equalizer to make a voice sound natural. I learned how to use a compressor to soften quick and loud sounds or to beef up thin sounds. It turned out that the return on investment for those hours was more than I ever hoped for!

I ended up favoring the simplest tools and I was very happy with the work I produced.

I believe the same principle can be applied to almost every aspect of life—finances, parenting, even software engineering.

As people, we naturally tend to look for flashy, popular solutions that promise to solve all of our problems (or at least to perfectly solve a single problem). We are misguided about these complex solutions. We’ve created complicated problems by not properly understanding the fundamentals of whatever we’re struggling with.

JavaScript Basics

We’ll be looking at basic programming concepts and how they can be applied to JavaScript. The goal here is to have code that is simpler, more flexible, easier to understand, and easier to test.

First, before introducing interfaces, I’d like to talk about a problem.

Mocking, stubbing, and mutating

Recently I was writing code that read files from the file system. The code worked great. To test it, I had to use a library that would stop my code from reading from the file system. My tests would have been too slow if I had let it do that. Plus I needed to simulate scenarios that would have been complicated to implement with the actual file system.

Historically I would have used a library like Proxyquire or Sinon. Proxyquire allows you to override the imports of a file. Sinon allows you to mutate methods on an object. You can use either or both of these to make your code easier to test. Although it would be better to use just one.

As an example, let’s pretend you have a module called “a”. Let’s also say that module “a” imports module “b”. Proxyquire works by importing module “a” and overwriting the exports of module “b”. It won’t affect other imports of module “b” elsewhere. Sinon works by mutating the exports of module “b”. It will affect every place that imports module “b”, so you must remember to restore it when you are done.

/* This is my file I'll be testing foo.js */

import fs from 'fs'
import { promisify } from 'util'

const readFileAsync = promisify(fs.readFile)

export function readJsonFile (filePath) {
 return readFileAsync(filePath).then(JSON.parse)
}

/* This is my test file foo.test.js */

import fs from 'fs'
import test from 'ava';
import { stub } from 'sinon'
import proxyquire from 'proxyquire'

test('readJsonFile with proxyquire', async function (t) {
  t.plan(2)

  /* fs.readFile is overwritten for this import of foo.js */
  const { readJsonFile } = proxyquire('./foo.js', {
    fs: {
      readFile(filePath, callback) {
        t.is(filePath, 'myTestFile')

        return callback(null, '{ success: true }')
      }
    }
  })

  const results = await readJsonFile('myTestFile')
  t.deepEqual(results, { success: true })
})

test('readJsonFile with sinon', async function (t) {
  t.plan(1)

  /* fs.readFile is overwritten everywhere */
  const fsStub = stub(fs, 'readFile')
    .withArgs('myTestFile')
    .callsArg(2, null, '{ success: true }')

  const results = await readJsonFile('myTestFile')
  t.deepEqual(results, { success: true })

  // Won't happen if test fails :(
  fsStub.restore()
})

Why are stubs bad?

Neither of these options is great because they involve mutation. In software development, we want to avoid mutation when possible. because mutation leads to a decrease in predictability across an application.

One small mutation never seems like a big deal. But when there are many small mutations it becomes difficult to track which function is changing what value and when each mutation is being done.

There’s also the nuisance of lock-in. Both sinon and proxyquire will require you to update your tests if you change your file system library from fs to fs-extra-promise. In both cases, you’ll still be using the function readFileAsync. However, sinon and proxyquire will keep on trying to override fs.readFile.

What are the alternatives?

To solve this problem I followed a principle called Dependency Inversion. Instead of my module creating its dependencies, it will expect to be given its dependencies. This produces modules that are both easier to test and more flexible. They can also be made to work with many implementations of the same dependencies.

/* This is my file I'll be testing foo.js */

export default function ({ readFileAsync }) {
  return {
    readJsonFile (filePath) {
     return readFileAsync(filePath).then(JSON.parse)
    }
  }
}

/* This is my test file foo.test.js */

import test from 'ava'

import foo from './foo'

test('foo with dependency inversion', function (t) {
  t.plan(2)

  const dependencies = {
    readFileAsync(filePath) {
      t.is(filePath, 'bar')

      return Promise.resolve('{ success: true '})
    }
  }

  const result = await foo(dependencies).readJsonFile('bar')
  t.deepEqual(result, { success: true })
})

Not only have precious lines been saved in our code, but there is also no more worrisome mutation happening! The module will now accept readFileAsync rather than creating that function itself. The module is better because it’s more focused and has fewer responsibilities.

Where does the dependency go?

The dependencies have to be imported somewhere. In an application that follows dependency inversion, you should move the dependencies as far “out” as you can. Preferably you’d import them one time at the entry point of the application.

/* json.js */

export default function ({ readFileAsync, writeFileAsync }) {
  return {
    readJsonFile(fileName) {
      return readFileAsync(`${fileName}.json`).then(JSON.parse) 
    },
    writeJsonFile(filePath, fileContent) {
      return writeFileAsync(filePath, JSON.stringify(fileContent)) 
    }
  }
}

/* content.js */

export default function ({ readJsonFile, writeJsonFile }) {
  return {
     getContent(contentName) {
      // business logic goes here.
      return readJsonFile(contentName)
     },
     writeContent(contentName, contentText) {
      // business logic goes here
      return writeJsonFile(contentName, contentText) 
     }
  }
}

/* index.js where the app starts */

import fs from 'fs-extra-promise'
import jsonInterface from './json'
import contentInterface from './content'

const json = jsonInterface(fs)
const content = contentInterface(json)

// content can be used by an http server
// or just exported if this is a library
export default content

In the example, you saw that the dependencies were moved to the entry point of the application. Everything except index.js accepted an interface. This causes the application to be flexible, easy to change, and easy to test.

What else can Dependency Inversion do?

Now that you’ve fallen in love with dependency inversion I’d like to introduce you to some more of its power.

When your module accepts an interface, you can use that module with multiple implementations of that interface. This is a scenario where the libraries TypeScript and Flow can be useful. They’ll check that you’ve provided the correct interface.

An interface is simply a collection of methods and properties. So by saying that a module accepts an interface, I am saying that a module accepts an object that implements a set of methods and properties. The expectation is that the interfaces similarly implement different functionality.

A common interface you might know is the React component interface. In TypeScript it might look like this:

interface ComponentLifecycle {
      constructor(props: Object);
      componentDidMount?(): void;
      shouldComponentUpdate?(nextProps: Object, nextState: Object, nextContext: any): boolean;
      componentWillUnmount?(): void;
      componentDidCatch?(error: Error, errorInfo: ErrorInfo): void;
      setState(
          state: ((prevState: Object, props: Object) => Object,
          callback?: () => void
      ): void;
      render(): Object | null;
      state: Object;
  }

Please don’t despair if you didn’t understand everything in that interface. The point is that a React Component has a predictable set of methods and properties that can be used to make many different components.

We are now beginning to venture into the territory of the Open-Closed Principle. It states that our software should be open for extension but closed for modification. This may sound very familiar to you if you’ve been building software with frameworks like Angular, or React. They provide a common interface that you extend to build your software.

Now, instead of relying on third-party interfaces for everything, you can begin to rely on your internal interfaces to create your software.

If you are writing a CRUD (create, read, update, delete) application, you can create an interface that provides the building blocks for your actions. Your modules can extend that interface to implement the business logic and use-cases.

If you are writing an application that performs tasks, you can build a task interface that provides the building blocks for different tasks. Each task can accept that task interface and extend it.

Dependency inversion and the Open-Closed principle allow you to write more reusable, testable, and predictable software. You’ll no longer have a jumbled mess of spaghetti code. Instead, you’ll have a uniform group of modules that follow the same pattern.

Many Implementations

There’s one more benefit to accepting an interface. You can implement that interface in many different ways.

Here’s my favorite example of this. Imagine that you have an interface for a CRUD application. You could have one interface that implements the database storage. This is great, but what if the database reads or writes become slow? You could also write a faster implementation that uses Redis or Memcached to speed up the response times. The only change you’ll have to make is writing a new interface. There will be no need to update business logic or anything else.

You could consider React and React-Native to be popular examples of this. They both use the same React component and React DOM interfaces, but they implement them differently. Even inside React Native, there is an implementation for both IOS and Android. Multiple implementations allow you to write your logic once and execute it in multiple ways.

Now what?

Now that you’ve learned about dependency inversion and the open-closed principle, it’s time for you to go and apply it in your code. Don’t write any imports in the next module you write. Instead, allow it to accept an interface. In your tests, you’ll be able to avoid third-party libraries that mutate your dependencies! Then try to start identifying where common interfaces can be used. You’ll slowly but surely create a better application!

This is a repost, the post originally appeared on www.justindfuller.com.

Hi, I’m Justin Fuller. I’m so glad you read my post! I need to let you know that everything I’ve written here is my own opinion and is not intended to represent my employer in any way. All code samples are my own and are completely unrelated to my employer's code.

I’d also love to hear from you, please feel free to connect with me on Github or Twitter. Thanks again for reading!

How To Write Error Messages That Don’t Suck

Justin Fuller — Mon, 02 Sep 2019 01:11:01 +0000

This post originally appeared on JustinDFuller.com.

“A validation error occurred.” Yep. Thanks!

The release is imminent; this is the last update that needs to be verified, and I get an error message that’s as useful as the close button on an elevator.

It turns out that it is a validation error, kind of. The input I am giving is a duplicate. It’s valid, it just already exists!

Wouldn’t it be so helpful to know that?

It would be helpful to be informed of several things when errors happen. I don’t need to know everything. The history of programming won’t help me here. The message should give me just enough info for me to get this error fixed so that I can finish my work, go home, and play with my kids.

What makes an Error?

In JavaScript, an error object always has the name, message, and stack properties. The name gives you an at-a-glance classification of the error. The stack tells you where it happened. The message? Well, according to some developers, you don’t need to worry about that! “The stack trace gives you everything you need.”

Please, don’t be one of those developers.

Useful Error Messages

Raise your right hand, place your left hand on a copy of “Clean Code” and repeat after me.

“I swear to include enough detail in my error messages that future developers will easily be able to determine what went wrong and what they need to do to fix it.”

What Happened

When a police officer pulls you over to give you a ticket, does it say “Bad Driving”? No! It says you were going 65 miles per hour in a school zone, you passed a stopped bus, and your car hasn’t been inspected in four years! Sure, you’re going to jail, but at least you know why.

So the error message from earlier should not be, “A validation error occurred”, but instead:

Unable to save model "user" because the property "email" with value "JustinFuller@company.com" already exists.

Instead of a simple error that says, “Invalid option” use:

The option "update" is not valid. Valid options include "upsert", "get", and "delete".

These updated error messages attempt to help us understand the cause, giving us a start toward the solution.

How It Might Have Happened

Now that the error is describing what exactly went wrong, it’s time to help the poor soul who stumbled into this predicament begin to climb back out. Watch carefully. When done correctly it will seem like you’re reaching forward through time by anticipating what could have led to this unfortunate turn of events. You’ll be right there with that future developer — maybe yourself — telling them that everything is fine, that you’ll get through this together.

You’ll begin by explaining what happened.

For anything that has a prerequisite step, such as configuration or validation, you can suggest verifying that step has been completed. Don’t worry if your error messages get long. It’s better to provide too much information than not enough.

I’ll add more detail to one of the earlier examples:

The option “update” is not valid. Valid options include “upsert”, “get”, and “delete”. **If you expected “update” to be an option, you must first export it from the file: "./src/controllers/index.js".**

Now you are anticipating how this might have happened: the developer probably just forgot to export the new option. The error becomes a reminder of that step. You’ve now shown two possible causes of the error; the first is a possible typo (here are the valid options) and the second is a configuration error (here’s where it should be exported).

The React library does an excellent job of anticipating how errors might have occurred. They don’t address every edge case, but they do give helpful hints for the most common errors. For example, you can’t use the function reactDom.renderToNodeString() in the browser because node streams don’t exist there. So React gives you a suggestion of how it happened and how to fix it:

ReactDOMServer.renderToNodeStream(): The streaming API is not available in the browser. Use ReactDOMServer.renderToString() instead.

There could be other ways for that error to occur, but they guess that the most common reason is that renderToNodeStream was called in the browser.

Relevant Data

While writing error messages you must remember that applications rarely do only one thing at a time. So when errors occur it’s a challenge to find the state of the application at that time. For this reason, it’s very important to capture the relevant data and relay it back to the developer so that they can pinpoint the cause.

In the first example, I included the phrase:

The property “email” with value “JustinFuller@company.com” already exists.

This is very useful but could be impractical. It may take too much effort or time to create natural language errors for every variation of data, or in some cases, we may simply be passing on a failure outside of our control, so the only option left is to give a good description and include as much relevant data that is safe to print.

The choice of what data is safe to print is tricky: if you choose exactly what properties to include then you end up modifying the list every time there is a new property, or, worse, you forget and it doesn’t show up when needed; on the other hand you can remove properties that are known to be unsafe, but here you risk adding a new property and forgetting to exclude it, causing leaked sensitive data. You have to use your judgment and consider your company rules. Does your software handle highly valuable or personal data that should not be written to a non-encrypted destination? Thoughtlessly logging out every object that causes an error is liable to get you fired from some jobs, while at others it’s the standard operating procedure. So please, use common sense and be careful!

Unexpected Errors

There are two ways to include relevant data when you don’t know exactly which property or action caused the error.

The first should be used when you intend for the error to be read by humans, you put the data right in the message: An error was received: “Duplicate entry found for user.email”, while upserting user: { “email”: “justinfuller@company.com” }. This error style has its drawbacks, like the entire object being placed into the error message that could be sent somewhere unintended. If, however, you know it’s safe then this style has the advantage of giving complete details about the situation.

In other cases, you may not want the data to leak to a log file or an API response. You can provide a reference ID, a timestamp that can be manually or automatically referenced to data later, or some other property that will allow the developer to track down the pesky data-point that caused an error.

Expected Errors

I’ll be the first to admit that I’m prone to making simple mistakes. I type “upswert” instead of “upsert”; I type “npm tes” instead of “npm test”. So it’s refreshing when I get an error message saying:

Received an unknown command, "npm tes". Did you mean "npm test"?

When the developers prepped for this, they must have gazed into the future and saw that someone would make that typo — or maybe they just know that humans are prone to silly errors.

Whenever there’s a step in a process that can go predictably wrong you have an opportunity to prepare clear guidance on what went wrong and how to fix it.

Steps To Fix The Issue

For some errors, it will be possible to give a solution to the error instead of just reporting that it happened. Some times it will be easy, like earlier when we showed the correct options after accidentally typing “update” instead of “upsert”. Other times you’ll need to put in more effort to give the user enough information to correct their mistake, like if you’ve detected a recursive dependency, and you need to tell them where the loop is and what they must do to remove it.

An error that doesn’t suck

So, do you want to provide helpful error messages? Next time you write an error, try to include a full description of what happened, how it might have happened, any relevant data that is safe to include, and any steps that might help resolve the problem.

This is a repost, the post originally appeared on www.justindfuller.com.

I’d also love to hear from you, please feel free to connect with me on Github or Twitter. Thanks again for reading!

Refactoring — oops, I’ve been doing it wrong.

Justin Fuller — Sun, 01 Sep 2019 19:07:54 +0000

This post originally appeared on JustinDFuller.com.

Welcome to my intervention. I’m a refactoring addict and I’m not afraid to admit it, but there’s only one problem: I’ve been doing it backward. You see, what I’ve been doing could be more accurately described as premature code abstraction.

We all know about refactoring. If you’ve read even a single programming book, or if you spend much time reading code blogs, you’ll have heard all about it. It’s an important concept that keeps code understandable, maintainable, and extensible.

At least that’s what everyone tells me.

So why has refactoring not been accomplishing what I was hoping?

As I wrote my most recent library, I took some time to reflect on the evolution of my code. I realized that before I had a fully working product and before I had an ideal output in my unit tests, I had refactored my code into interfaces that I wasn’t even sure I would need. I had moved code around, made it extensible, made it reusable, but why? Was that code going to give me the final output I needed? I didn’t know yet.

Everything worked out in the end, but was my code more complicated than it needed to be? I believe so.

Principles Over Purpose

Have you heard of SOLID principles? I try to follow them closely. Every function that I write aims to have a single responsibility. My classes and factories aim to be open for extension while discouraging modification. I also try not to depend directly on too many things, so instead, I accept dependencies as arguments in functions and classes.

Does that like a recipe for good code? I think it does. The problem occurs when my code focuses on being SOLID, or pure, rather than on accomplishing what it was born to do. The problem occurs when I put principles over purpose.

For example, I’ve been so focused on making sure my unit tests have no expensive IO (input and output). I’ve occasionally had to go back and fix code that was wrong due to my incorrectly mocked dependencies.

So, what’s the solution?

Remember that reflection I mentioned earlier? It reminded me of the mantra, “Make it work, make it right, make it fast.” I’ve realized I’ve been going out of order. I’ve been making it right, making it fast, then making it work!

Make It Work

As I’ve begun to write more it has become clear that good writing doesn’t just happen. First I have to get all my thoughts down on the page. I have to see where my thoughts take me. Then I must shape them into some sort of semi-coherent and non-rambling version of what just spilled out.

The same thing can happen with code.

Get it all out there into that function. At first don’t worry too much about naming, single responsibility, or being extensible — you’ll address that once your function is working. To be clear, you won’t be writing your whole application like this, just one small piece.

Once you’ve got the output you are looking for (you’ve got unit tests to prove that the code is correct, right?) begin refactoring, but don’t go too far too fast! For now, stick with refactoring strategies that are in the category of proper naming, functions doing only one thing, and the avoidance of mutation; don’t immediately start making extensible or reusable classes and factories until you have identified a repeating pattern.

At this point, it makes sense to use any refactoring that has a logical benefit. This means refactoring with the purpose of the code being understood, or the code being reliable.

Consider postponing refactoring with patterns that are only useful in certain scenarios.

You’ll want to save those until you have a reason.

Have A Reason

Having SOLID code is not a reason. Having functional or pure code is not a reason.

Why do we make our code extensible? So that similar, but not identical, functionality can branch off of base logic.

Why do we invert dependencies? So that the business logic can be used by multiple implementations.

Hopefully, you see where I am going with this. Some refactoring stands on its own. For example, refactoring the name of a variable to become more accurate will always make sense. Its merit is inherent. Refactoring a function to be pure usually makes sense because side-effects can cause unforeseen issues. That's a valid reason.

“It’s best practice to use dependency inversion” is not a reason. “Good code is extensible” is not a reason. What if I only have a couple of never-changing dependencies? Do I still need dependency inversion? Perhaps not yet. What if nothing needs to extend my code and I have no plans for anything to do so? Should my code increase its complexity just to check off this box? No!

Take a look at the following example.

// not extensible

function getUser() {
  return {
    name: 'Justin',
    email: 'justinfuller@email.com',
    entitlements: ['global', 'feature_specific']
  }
}

// used later

getUser().entitlements.includes['feature_specific']

// Extensible

class User {
  constructor() {
    // initialize here
  }

  hasEntitlement(expectedEntitlement) {
    return this.entitlements.includes(expectedEntitlement)
  }
}

// used later

new User().hasEntitlement('feature_specific')

Which do you prefer? Which do you naturally tend to write first? Of course, the User class is far more extensible because it can be overriden by another class. For example, if you had a SuperUser then you could implement hasEntitlement like this:

hasEntitlement() {
  return true
}

Don't let the Class throw you off. The same result can be accomplished without it.

function superUser(user) {
  return {
    ...user,
    hasEntitlement() {
      return true
    }
  }
}

Either way, this encapsulation of hasEntitlement allows the User to, for different use cases, take advantage of polymorphism to extend—rather than change—the code.

Still, that User class may be complete overkill, and now your code is more complicated than it will ever need to be.

My advice is to stick with the simplest possible pattern until you have a reason for something more complex. In the above solution, you may choose to stick with the same simple User data object until you have multiple user types.

Order Of Complexity

And now, if you’ll allow it, I’m going to make something up! I call it the order of complexity and it helps me when I make refactoring decisions. It looks like this:

Constant Variable
Mutable Variable
Collection (Object, Array)
Function
Function with Closure
Factory (A function that returns a collection)
Class

Whenever I decide how to organize functionality, I refer to the list. I choose the highest possible choice that will suffice for my implementation. I don’t choose again until it simply will not work. Sometimes performance will affect this choice, but not often.

Usually, I find that I’ll put something in an object instead of a simpler constant variable. Or I created a factory when I only require a function.

This list keeps me grounded. It prevents me from prematurely refactoring.

Balance

I recently heard that if you say in a meeting, “it’s all about finding balance,” everyone will nod their head at your meaningless comment like you’ve said something profound. I’ve got to give it a try soon.

Here, though, I think balance is important. As programmers, we have to balance code quality, performance, maintainability, with the good old-fashioned need to get things done.

We have to be vigilant and make sure both needs stay in their correct place. Our code can’t be maintainable if it doesn’t work correctly. On the other hand, it’s hard to make bad code work correctly.

Still, code may be refactored, but what if it’s been refactored past the point of what is useful? These are important questions to keep in mind.

Next time you write your code, please, refactor! But also, maybe… don’t?

This is a repost, the post originally appeared on www.justindfuller.com.

I’d also love to hear from you, please feel free to connect with me on Github or Twitter. Thanks again for reading!

Service calls make your tests better

Justin Fuller — Fri, 19 Jul 2019 02:13:48 +0000

This post originally appeared on JustinDFuller.com.

TL;DR: If all tests are mocked, you don’t know if your code really works, you only know that, theoretically, it is supposed to work if the integrations adhere to the contract you expect.

Mocking, stubbing, or maybe—even better—dependency inversion, they can simplify testing and make your code easier to change, but can they cause problems too? Let’s see.

Take a look at this test where we are saving a file using an external file service.

Can you determine if this is a useful test?

The answer is that it most likely isn’t going to ever catch any bugs. Let’s see if we can determine why not.

The first reason is that we are using a mock implementation of the drive service. So we won’t catch any errors if someone changes the drive service without changing the file uploader.

Now, what about even further down the line? What if something changes with the actual drive server that we want to integrate with? We certainly won’t catch those errors because we aren’t ever making a call to it.

In the end the only thing we are truly testing is that the uploadFile method gives the file object to the driveService’s write function. We’ll catch an error if drive service uses the file object incorrectly, or if fileUploader stops giving the file directly to the drive service.

Unfortunately we will also have to update this test whenever we change how fileUploader and driveService interact.

So not only is the test brittle, it’s just not very useful. Further, in a typed language it would be completely useless since the compiler would catch these types of errors during compilation.

So, how can we make this test better? The test can become useful if it actually makes the service call to the drive server. Not the internal drive service object, but to a real drive server.

Immediately you should be objecting, “My unit tests will take forever to run and become extremely brittle if I am making real service calls!” If you said that then you are absolutely right. This test is best served as an integration test.

Integration Tests

Integration tests may not be run as often as unit tests, but they should—at the very least—be run before integrating your changes into the code base. See what I did there? Integration tests run when you integrate.

Still, the problem exists, how could I realistically run all the servers needed for my application to function? Starting them up can be expensive, not to mention the data stores, managing ports, authentication, and everything else that goes into creating a fully-functioning system.

Take a look at this simple system diagram. It represents a very stripped-down version of the system in the example test from earlier.

You can see that all we’re interested in here is testing the integration of our “File Service” with the external “Drive Service” that belongs to another team. In this case we’re not trying to run a full end to end test.

But what are we actually testing?

Oops! Here, as shown in green, only the file service was tested. We wanted to test if our File Service and its connection to the Drive service are actually working. So instead of writing a mocked version on our end, we will look for a way to obtain a testable version of the drive service.

Isolated Services

One option is to create an isolated version of the Drive Service. Ideally this will be owned by the team that created the Drive Service. In order to ensure that the fake server is trustworthy they will run the same tests against both the fake server and the real server.

The above code represents a sample isolated server implementation. You can see that it is very similar to the real server implementation, except that it uses an in-memory data store instead of a real file storage device. It even uses port 0 to ensure that an ephemeral port is used, further increasing the stability of it’s tests.

Now that the Drive team is providing this isolated server, our integration tests can safely start it and use it during integration tests. Let’s rewrite that original test as an integration test and see if it becomes more useful.

Now, is our test more useful? Since we made the call to the real Drive Server api (even though it saves to a different storage device, the API and business logic remain the same) we will know if our integration breaks.

Even better, not only were we able to test what URL it returns, we were able to test if the content was saved as expected. Our test will actually tell us if file saving works!

Look at our system diagram again. You can see in green the services that are being tested. This time we are testing the file service, the drive service, and, most importantly, the connection between them.

It isn’t realistic to expect that this set of tests will be run every time we change a line of code—that expectation is reserved for unit tests—but this test is still lightweight enough to run on each code check-in. Executing the integration test in this fashion will ensure that your main branch(es) will have not only correct business logic, but also working integrations with other services.

A fallback when an isolated service just isn’t possible

Sometimes you really may have some kind of environment or build-time constraint that makes an isolated server impossible. If that’s the case then you could look to fake APIs as a fallback.

Remember, we’re still talking about integrations here—code that interacts with other services. You may have noticed the code contained two Drive related entities: “DriveServer” and “DriveService”. The DriveServer was the actual service that belongs to the third party. We were using their in-memory version to test our integration with their service. The DriveService is an API that knows how to interact with a DriveServer. This API also belongs to the Drive team.

Thankfully they understand that not everyone can use their isolated in-memory server, so they also created a fake version of their API. Take a look.

This FakeDriveService is an implementation that the Drive Team could provide to anyone who uses their service. They are saying “If you test with the FakeDriveService you can trust that the real DriveService will work. We run tests against both to ensure that they work the same.”

This implementation is obviously even lighter than the isolated server, so what’s the drawback? Let’s refer, once again, to our system diagram.

While we are technically testing the connection mechanism we aren’t actually touching the Drive Service. Our test is built on trust, not actual verification. The trust is that the fake service really does work the same as the full service. In many scenarios this could be good enough, but if you have a production critical system you may need a better guarantee.

Still, this test is better than the original mock function that we started with. That mock function was completely untrustworthy to the point that, I would argue, it actually tricked us into thinking we were safe from bugs, but in reality we had no idea. Our tests will now have to change if the Drive Service changes. In that first mock scenario we wouldn’t have had to change our tests, it would have tricked us into thinking that our code still worked, even when it was broken due to an API change.

Credits

This post was directly inspired by posts that I recently found in Google’s “Testing on the toilet” blog. I wanted to take their idea and reframe it in the lens of a JavaScript application. Please see the links below to read their original posts.

Disclaimer

The views and suggestions here are my own, not my employer’s. I in no way intend to represent them through this post.

Get in touch

I’d love to hear from you. Please feel free to reach out to me on Github or Twitter.

This is a repost, the post originally appeared on www.justindfuller.com.

DEV Community: Justin Fuller

Twilight of the Software Engineering Team

The Dawn of the Agentic Team

Caveats

Keeping One Tab Open

Write It Down

Trello

Step 1: Record All Inputs

Step 2: Add Detail

Step 3: Focus

Switching Tasks

Finishing Tasks

Cleaning Up

Putting it All Together

Why do we fall into the rewrite trap?

My History

Cold–what?

Taking advantage of new technologies

Contempt Culture

The rewrite

When to Refactor

If your reason for rewriting the code is that you don't understand it, you should not rewrite it.

Hurdles to Refactoring

Backward Refactoring

When to rewrite

Rewrite for better functionality, rather than better code.

Key Takeaways

Go Things I Love: Channels and Goroutines

First Class

Communicating by sharing memory (👎)

Share memory by communicating (👍)

Timeout

Deterministic Randomness (See: oxymorons)

Another send-only channel

A read-only channel

The Main Function

Final Thoughts

Go Things I Love: Methods On Any Type

Methods

Encapsulation

Using a custom type

More encapsulation, delegation.

Keep a git repository of all your practice code

My knowledge repo

Your knowledge repo

Simply JavaScript: a straightforward intro to Mocking, Stubbing, and Interfaces

JavaScript Basics

Mocking, stubbing, and mutating

Why are stubs bad?

What are the alternatives?

Where does the dependency go?

What else can Dependency Inversion do?

Many Implementations

Now what?

How To Write Error Messages That Don’t Suck

What makes an Error?

Useful Error Messages

What Happened

How It Might Have Happened

Relevant Data

Unexpected Errors

Expected Errors

Steps To Fix The Issue

An error that doesn’t suck

Refactoring — oops, I’ve been doing it wrong.

Principles Over Purpose

Make It Work

Have A Reason

Order Of Complexity

Balance

Service calls make your tests better

Integration Tests

Isolated Services

A fallback when an isolated service just isn’t possible

Credits

Exercise service call

Fakes

Hermetic Servers

Disclaimer

Get in touch