Performing Rapid Code Rollbacks Using Configuration

Moran Danieli-Cohen — Tue, 28 Mar 2017 19:03:42 +0000

Originally published at msscohen.com - a mind of a software engineer

A common code release looks like this:

System before change consists of three components

System after change: One component is replaced with an improved component.

If something goes wrong and and a rollback is possible, the rollback should replace the new production code with the old production code (replace C2' with C2). The rollback plan usually means:

Revert the current release by committing the opposite diff to the project's version control.
Deploy the above change to production.
Restart the project's services to apply the new code.
Once the rollback process is completed, additional fixes may be needed such as repairing damaged data.

The rollback process may seem trivial but when working with distributed systems, performing a commit and releasing a new version to dozens of servers may take time. This time is valuable since it is the time where a buggy production version is working which may result in more corrupted data, more data loss, additional downtime and so on.

One way to handle this situation is by applying an architectural pattern which will allow rollbacks by changing a single parameter in the project's configuration file. The following is a diagram of such architecture:

Architectural pattern adjusted for rapid rollbacks: data is copied and distributed to the old and new components using a Hub; The Switch component decides which path to enable using corresponding configuration keys.

This architecture presents two new components:

Hub: Copies and distributes traffic to several outputs. For file systems, this component will copy files to several directories; For messaging queues, this component will route messages to multiple queues (for example by using a fanout exchange); For HTTP servers, this component will copy the HTTP request and send it to multiple destinations.

Switch: Choose which source of traffic to output according to configuration. You can either implement a generic switch or apply a switch per endpoint which enables or disables each endpoint's output according to a dedicated configuration key.

One of the advantages of this pattern is that you can deploy the new code without using it (when the switch is off), so you can test the code with real production data and stress, and activate it with a simple configuration change. Also, the hub and the switch are generic components which may be used for additional use cases.

A disadvantage of this pattern is that for small releases writing a hub, writing a switch and duplicating all incoming data might be overkill. Another disadvantage is that once the new release is stable enough, the next release should delete the of old version (C2, and possibly the hub and the switch).

I apply this pattern for either high risk changes or for large releases. In one of the biggest projects I managed, I kept the new version in production for a few months without enabling it using this pattern. During these months, we added more and more components and tested the system's behavior with live production data without affecting customers. Only when we were confident that our system was reliable, we enabled it.

3 Effective Ways to Maintain High Energy Levels at Work for Software Engineers

Moran Danieli-Cohen — Mon, 06 Feb 2017 18:56:21 +0000

Originally posted on msscohen.com - career hacks for software engineers

Although software engineering is often considers a dream profession, in real life it is sisyphean work which often requires many hours at the office under the fluorescents lamps, a freezing air conditioner and no sunlight. On the other hand, writing simple code which works is a craftsmanship and like so it requires inspiration, focus and high energy levels. In this article I will try to demonstrate how to balance between the two and manage to maintain energy levels high in a demanding workplace.

1. Find Solutions for Energy Waste

I returned to work 5 months after having my first child. My baby was still waking up 2-3 times at night and I was exhausted. My fatigue levels were so high that I decided to stop driving to avoid accidents. Thinking about returning to work and managing to function normally seemed like an impossible mission. I decided to analyze my daily routine and see if I can avoid energy waste.

Knowing myself, my most productive time of the day is the morning. Therefore, in order to be effective I should make sure that my energy levels are high in the morning hours and are being used correctly.

Analysis of my morning hours showed that most of my energy is spent even before I arrived to the office. In those days I woke up at 5:00 am to feed my child, dressed him up, prepared myself, took him to his daycare and then took a 2 buses ride to get to work. When I arrived to work at 8:00 am, I was already exhausted.

I considered replacing the 2 buses ride with a cab. The price of a cab is expensive - about 10$ in each way. Meaning, 400$ a month. Once I realized that by paying for a cab I'm actually paying for a more productive work day, it seemed more reasonable. Looking at the total value instead of the price of an individual cab ride, I started taking a cab about everywhere - meetups, courses and conferences that I wouldn't go to otherwise.

To conclude: check what is your most productive time of the day, analyze where your energy is wasted in this time and find solutions for energy waste by concentrating on the big picture and not on an individual action.

2. Take a Real Break

One of the workers I know doesn't take breaks. None. He arrives early and leaves the office at 19:00. Sometimes he continues working from home too. There are days he doesn't go to lunch breaks either: He just grabs something and eats at his desk while he continues working or he rushes to the nearest restaurant, buys something to go and disappears back to the office. There are managers with similar behaviour.

Unintentionally, this behaviour creates stress, burnout and it is a killer antidote for creativity. Managers who act this way create the same level of expectations from their workers, even if they don't ask it explicitly.

One of the main evils of this profession is that in most companies workers are still measured by the time clock and not by performance. Recent studies show that breaks are good for the brain: they increase creativity, improve focus and keep us healthy since we're not designed to sit all day.

So in order to maintain energy levels high, next time you feel tired, not in focus or stuck with a bug for hours, continuing to work will only make you feel more tired. Try taking a break instead and the bug will be resolved before you notice.

The meaning of break is not simply stop working. The meaning of a real break is giving rest for your brain. During break time do something which will help calm your mind like getting some fresh air, listening to your favorite music, drinking green tea and water, stretching exercises, and taking a shower or a quick nap if possible.

3. Shift Between Different Tasks

The software engineering profession is full of tasks that require different skills: reading emails, writing code, performance testing, talking to stakeholders, learning a new topic, reviewing code, debugging an issue, writing documentation, designing a feature, etc.

If you're working on a task which requires one skill and feel like you are running out of energy, switching to a task which requires a different skill is similar to taking a break for a specific part of the brain. So you can switch to another task which gives the part of the brain that was working some rest and then go back to that task later.

For instance if you code for 2 hours and feel lack of energy, try talking to stakeholders or reading emails, which require different skills. If you learn a new topic and need some rest, try reviewing code or debugging an issue. After some time, go back to the original task and you'll notice increased energy levels.

To summarize, in order to maintain energy levels high at a demanding profession such as software engineering: find solutions for energy waste by concentrating on the big picture and not on an individual action, take a break and do something that will calm your mind, and shift between different tasks to allow a specific part of the brain to rest.

DEV Community: Moran Danieli-Cohen

Performing Rapid Code Rollbacks Using Configuration

Further Reading

3 Effective Ways to Maintain High Energy Levels at Work for Software Engineers

1. Find Solutions for Energy Waste

2. Take a Real Break

3. Shift Between Different Tasks