DEV Community: Jorge Rodríguez

Bugs will happen. Are you prepared?

Jorge Rodríguez — Thu, 02 Jun 2022 11:02:02 +0000

It's very common for developers (and managers!) to focus on new features, performance, or bug fixing when coding. But when a project is complex enough, there's a hard truth that we all must take into account: bugs will happen. Period. I have never seen a new project go live without issues. In fact, I don't think I've ever seen a project stay bug-free in production for more than a month straight. Software engineering is imperfect, and our code must be prepared to deal with it. This post is about what you can do to make life easier for you and your team.

A couple of years ago, a bunch of us at The Agile Monkeys started this huge project for a bank. We rebuilt a critical part of their system, with an event-driven architecture that would be processing millions of monetary (and non-monetary) transactions daily. The team did a fantastic job and it's still in production and working well to this day. However, it was not a bed of roses. We faced some important challenges that I believe made us all much better developers today.

As usual with any big project, there was a lot of technical learning involved. From do's and don'ts in event-driven architectures, to communication both inside and across the teams. There were loads of lessons learned. But there was one huge takeaway that I took from the project. We dedicated way too much time to code optimization, bug fixing, and new features. And we didn't work enough on defensive programming.

What's defensive programming?

Quoting Wikipedia directly:

Defensive programming is a form of defensive design intended to ensure the continuing function of a piece of software under unforeseen circumstances. Defensive programming practices are often used where high availability, safety, or security is needed.

The emphasis here is on unforeseen circumstances. Bad Things™ will happen, no matter what you do. So you have to prepare for them.

At our first launch, we thought that we had a very robust application. It was performant and well tested. But, again, Bad Things™ can and will happen. In our case, it was an unknown rate limitation on our vendor production API that temporarily shut our application account down. Then, even when our services had a retry mechanism in place, they all ended up retrying simultaneously. As a result, a large majority of our transactions failed. Recovering from those errors required a lot of work from all of us. We were clearly not prepared enough for an issue at such a scale. We found ourselves doing a lot of manual, tedious work to solve the transactions in pending status. It resulted in many hours of hard work, and it naturally grew into some automatic tooling for reconciliation. Tooling that we should have implemented right away, instead of dedicating that much time to optimize performance, fixing bugs or even adding new features.

Reconciliation tooling

In regards to the tooling that you should have in place, it will greatly depend on the needs of your system. But for an event-driven architecture like the one that we were working with, we found it incredibly useful to have an Admin API (be careful with the security on this one!), with some automatic checks and reconciliation logic. The methods in this API would be able to replay events, look for inconsistent values in DB, or for the most extreme scenarios, even update some DB fields directly.

That, combined with an event store where every event received is kept (especially if not completed successfully), will allow your team to work wonders. It's important that the developers are able to safely and consistently replay the events. One of the issues that we faced in our project was that, even though our log aggregator worked very well to grab event data, it was not consistent enough for our scale. A small percentage (below 1%) of events were lost, but it equated to thousands of events that need to be manually searched for. Having a more complete and reliable event store for our needs was a huge relief for our reconciliation process.

There's another very good benefit that you'll get from all of this tooling: you can automatize those checks and processes to run periodically once you've verified their robustness. For instance, you could run a reconciliation task daily on all of those events that failed or stalled for more than an hour. Once the tooling is in place, there are many advantages that you can enjoy!

So, for every reader out there, I'll be proud if you can just carry home the following takeaway: bugs will happen. Are you prepared to deal with them in production? If not, start dedicating some time to it, even before bug fixing or optimizing your code. Invest in a calm mind. Future You will thank you when enjoying the spare time with your friends instead of dealing with on-call issues!

Coronavirus can be the perfect opportunity to embrace remote work

Jorge Rodríguez — Fri, 13 Mar 2020 17:08:35 +0000

At The Agile Monkeys, we’ve been working remotely for 10 years. From the start, the founders of the company always challenged every business convention that did not make any sense to us. Spending a lot of hours in an office was one of those. We always thought that is important to be able to work focused and at the time and place that you find more productive. Working from home (or wherever you like) was a part of it. Now we see a lot of companies struggling to implement that philosophy, and chances are they’re struggling because they didn’t have the time or motivation to build a culture around it. Maybe the Coronavirus pandemic will help to change some misperceptions about remote working that we’ve been hearing for a very long time.

It is very common to see companies getting paralyzed if their employees cannot get to the office. This is often seen when there is extreme weather or even major traffic jams. Companies seem to just accept that paralysis, probably because it does not happen frequently enough. But a lockdown of a couple of weeks? Well, that’s a different game.

It is very easy to imagine why companies that embrace remote work have a competitive advantage. They will not stop working just because their employees can’t go to the office. They will not be forced to negotiate to extend deadlines or contracts with providers or clients, because they will keep working at full throttle. This is the kind of thing that seems so obvious from the inside. Yet the average company has typically refused to accept remote work.

It turns out that sometimes this kind of crisis is the boost needed to force these changes in society. Acme Corp. and the rest of the companies who are so proud of their open-plan offices will have to reconsider whether remote work is a good option for them. If for no other reason, they will have to, because while they are on hold, their competitors will be able to go on with their work.

It is true that not all jobs can be performed remotely, but it's also true that those would also benefit from a general shift to offsite work. Think of how traffic would be affected by removing all those commuters, not required physically anymore, from the roads. And I haven’t even mentioned the ecological benefits of such a decision.

Unfortunately, establishing a remote work policy for a company is not as easy as sending a company-wide e-mail (although that can be a good first step!). This kind of work requires the company and employees to think carefully about how they will alter their work habits. Everyone will need to change how they approach challenges and especially how communication and documentation are handled. It won’t be possible to just stand up and go over to your workmate to ask them about how they dealt with an issue in the past.

Often, that scenario is considered a drawback when remote work is being discussed. But what if we look at it from the other side? That workmate has just been saved from an interruption. Now multiply that by the number of times it happens in an average workday. That workmate is now perfectly able to focus on their work and, once they feel like they can stop and attend to other matters, they’ll get to your question. Or even better, they might just share the solution as an internal document so that no one gets blocked waiting for them to respond. This is why asynchronous communication works so well for remote environments.

As offsite contractors, we have worked with several top-notch companies and institutions like eBay, Harvard University, Rent the Runway, Zara, and many others, always with a think-remote-first approach. And we know that working remotely has been one of our secret sauces. When properly done, remote work can make you better: more productive, more efficient, happier, you name it. But to take advantage of it, you must start thinking as a remoter because, suddenly, documentation and communication are key aspects of your organization. And it will pay off.

From our beloved Canary Islands in the Atlantic Ocean, we have been able to work while our clients suffered from extreme weather. Or power outages. We can also continue working even if transport or traffic blocks our clients on the way to their headquarters. And we will keep doing it if they are forced to stop working due to a lockdown. We will always be there. And we hope the current worldwide difficulties will be the catalyst for other companies to experiment with remote work and discover its many business benefits.