DEV Community: Akshat Shah

Circuit Breaker - Hope is not a Design Method

Akshat Shah — Sun, 18 Jun 2023 11:45:10 +0000

TLDR: The Circuit Breaker pattern - fail fast when it's likely that an operation will fail based on its previous invocations.

Context

Say you're placing an order on a groceries app. You make the payment and you're waiting for that happy payment confirmation screen with its melodious chime that almost always appears immediately. However, this time, it does not. You wait thirty long, painful seconds and are eventually annoyed to see a "Could not confirm payment" message.

Now let's say this is what normally happens when you make a payment: the app on your phone polls an order service hosted somewhere remote, which in turn polls a dedicated payment confirmation service.

Let's say that the payment confirmation service is down due to some reason.

In a distributed system such as this, where there are multiple remote service calls to serve a transaction, failures can be broadly categorized into two types, based on their duration:

Transient failures: these are resolved automatically after a short time interval (seconds). Reasons for these may include timeouts due to slow network connections, high load on the server, or any temporary issue. These can be handled by retrying the call.
Failures that will take longer to fix: for example, the complete failure of a service or broken network connectivity. These are the type of failures that we'll look at.

What Happens When a Dependency Fails

Let's say we handle all failures of this frequently executed operation with a retry mechanism. The calling application (order service in the example above) will continually retry the remote call (a specified number of times). But we know that the retries will not succeed since the failure at the dependency (payment confirmation service in the above example) is not transient. What are the consequences?

The transaction is blocked on this operation. It is using up resources - memory, threads, database connections and CPU cycles that are involved with this transaction. This is especially impactful to the overall performance of the calling application if the number of transactions that involve the dependency is high in volume or frequency - like grocery orders!
Blocked transactions and holding resources wastefully will cause failures to cascade further to possibly unrelated parts of the system that might need to use the same resources. And this is not just at the calling service level where memory, threads, DB connections et cetera are being held. The service itself could be a resource to other remote components, which might also get affected since the calling service is not behaving as usual. In the grocery order payment example, resources on the user's phone are being used by the app polling the order service.
Continuous retries to the already failing dependency might put further load on it and perhaps prevent it from recovering
And, in the example above, you have a lot of annoyed users that waited a long thirty seconds!

Great, Now What?

"Hope is not a design method."

~ Michael T. Nygard, Release It!: Design and Deploy Production-Ready Software

The key point to note here is that non-transient failures of a dependency shouldn't be handled the same way as transient failures. We need a way of detecting that a failure is non-transient and acting accordingly. Another thing to note is that there's nothing special about a remote service call here; any operation that is going to fail for a while can benefit from such a preventive arrangement. Let's think through what we could do in such a scenario and lay down some requirements:

Detect that the failing operation is probably going to keep failing for a while - it's not a transient failure
Fail fast - don't waste resources attempting the operation; skip that operation and respond immediately with an appropriate exception when it's known that it's likely to fail for a while
We need to eventually switch back to calling the operation. So, we need to detect if the problem has been fixed, and switch back when we are confident enough

Do we really need something like this? Why not just use fewer retries or a shorter timeout if we are worried about wasting resources and burdening services?
Valid question! The problem with setting shorter timeouts is false negatives. If we reduce the timeout duration, the operation might fail most of the time even though it could succeed given some more time. And for a high-volume transaction, wasted resources and their second-order consequences due to even a few retries can be significant.

Enter the Circuit Breaker

The circuit breaker pattern, popularized by Michael T. Nygard in his book, "Release It!", is what we can use in such a situation.

What it Does

Simply said, a circuit breaker acts as a proxy for the operation that might fail. This proxy monitors the number of recent failures and decides whether to attempt the operation next time it is triggered or return an exception directly, skipping the operation. It tries the operation after some time and decides whether to switch back to invoking the operation normally that it is acting as a proxy for.

How the failing operation recovers is handled externally, possibly by restoring or restarting a failed component or repairing a network connection.

States of the Circuit Breaker

Overall, we can guess that the circuit breaker has three states. And the naming of these states is where the "circuit" analogy comes into play:

Closed: The circuit is closed, so the request to the circuit breaker is routed to the operation which returns a response - the normal flow, so to speak. The operation can fail here, and the circuit breaker tracks these failures
Open: The circuit is open, which means the request to the circuit breaker will not be routed to the operation. Instead, the circuit breaker will return an exception
Half-open: This is the state in which the circuit breaker is 'testing the water'; it allows a limited number of requests it receives to invoke the operation, to check whether the problem with the operation has been fixed

State transitions

Let's look at how the circuit breaker goes from one state to another:

Closed -> Open:
- While in the closed state, the circuit breaker maintains a count of the recent failures in invoking the operation
- If this count (with recency kept in consideration) exceeds the failureThreshold, the circuit breaker transitions to the open state
Open -> Half-Open:
- While in the open state, a timer is set for breakDuration
- During this time, the operation is not invoked at all, and the circuit breaker responds with an exception. This is also the time during which the fault is expected to be fixed
- When the timer expires, the circuit breaker switches to the half-open state
Half-Open -> Open:
- Implementations can differ here, but here's one of the simpler ones: the next requests to the circuit breaker when it's in the half-open state are routed to the operation. If these invocations succeed, depending on the successThreshold, the circuit breaker will eventually go to the closed state
- However, if there is any failure before successThreshold is reached, the circuit breaker reverts to the open state
- Another implementation might allow a fraction of the incoming requests to invoke the operation to decide whether to switch to the closed state. This can be helpful, for example, if a remote service being called is recovering and we wouldn't want to flood it immediately with the regular volume of requests.

Usage

In code, the circuit breaker acts as a wrapper around the operation under consideration. So, any calls to the said operation go through the circuit breaker proxy.

Parameters

Different implementations require different parameters to be configured, but here are the core ones:

failureThreshold: The threshold (count or percentage) for failures after which the circuit breaks. Some advanced implementations also couple this with a minimumThroughput, so the failure threshold only takes effect when there's a minimum amount of traffic
breakDuration: This specifies how long the circuit remains open before switching to the half-open state. Too low and there will be more failed invocations, while too high would mean a lot of failed transactions unless there's a fallback
successThreshold: The threshold (count or percentage) for successful invocations when in the half-open state for switching to the closed state

These values should be tuned by looking at existing traffic patterns and expected recovery time for non-transient failures. It's difficult trying to estimate optimum values without this information.

Extras

Handling the Exception When the Circuit is Open

In the open state, the circuit breaker will return an exception. This exception can be handled according to the application. Here are some options:

Report it appropriately to the user, asking them to try later. In the grocery app example, perhaps informing the user immediately that there was a problem in confirming their payment might be a little less annoying than making them wait for... thirty long seconds (okay, last time!)
Try an alternative operati****on, like a fallback. Perhaps using a different way to confirm the payment, or users could be shown alternative payment options on their app
Temporarily degrade the functionality of the system proactively instead of users getting an error at all, like making a payment method unavailable while the circuit is open

Break the Circuit Based on the Type of Exception

Most circuit breaker implementations allow the specification of the types of exceptions or results that should be considered when breaking a circuit. For example, exceptions arising from valid business logic conditions can be ignored by the circuit breaker while still tracking service unavailability exceptions. In the case of HTTP calls, perhaps some 4xx response codes (client error) could be ignored while 5xx codes could be tracked.

Log Circuit Events

Circuit breakers are a valuable place for monitoring. Any change in breaker state should be logged and breakers should reveal details of their state for deeper monitoring. Breaker behaviour is often a good source of warnings about deeper troubles in the environment.

~ Martin Fowler, in his blog about circuit breakers

As an example, the circuit breakers offered by Polly and resilience4j provide hooks into their state transitions, where you could log useful information. These logs could be tied to alerts for monitoring and observability.

Conclusion

Circuit breaker is a pattern to prevent an application from invoking an operation if it is highly likely to fail. This helps save resources, offer better response times and increase the stability of a system when it is in an error state or recovering from one. However, it's not the best one to use in all cases. Using a circuit breaker isn't recommended for handling access to local private resources in an application like in-memory data structures, since it will add a lot of overhead in this case. As mentioned above, retries are more suitable for transient failures. In fact, circuit breakers can be combined with retries to counter both types. It can also not be used as a substitute for handling business logic exceptions - it's a fault tolerance and resilience pattern.

References and Further Reading

Project Pandemonium

Akshat Shah — Fri, 14 Jan 2022 12:04:29 +0000

This blog was originally published by me as part of a blog series by Google Student Developer Clubs, VJTI.

A few days ago, during a lazy afternoon, while rummaging through my muddle of a cupboard, I found a notebook that had, unsurprisingly, my name scrawled on it. I opened the book somewhere in the middle, but those pages were blank. Then, opening the first page, I smiled, sheepish and nostalgic. Written on the first twenty or so pages of this notebook, was the account of one of my first projects as an engineering student.

I immediately took a trip down memory lane. I perused those logs, consisting of long paragraphs of technical details, diagrams and outlines of activities carried out each day. But what stood out were the number of questions and exclamations. Occasionally, a few lines lamented on the roadblocks and failures that were faced, the uncertainty of whether things will work out and whether we were getting anywhere with the project.

About the project

Along with three friends, I was part of a project that aimed to develop a WiFi Positioning System, or more specifically, an Indoor Positioning System. Fundamentally, this system would determine the location of a device using wireless access points that are in its vicinity. This project was under Eklavya, a mentorship program of the Society of Robotics and Automation, VJTI.

I would have loved to tell you that despite the misgivings in my memoirs, we managed to accomplish our goals. We didn't. And that's what this blog is about.

A project can be very chaotic. A number of things can go wrong, and every action to correct those mistakes can cascade into a wave of defects. In the end (if there is one), you are left with a good solution, a bad solution, a good solution that is not required, or several other variants. I'm no software engineering expert or a project management expert. I'm a final year undergraduate and I want to share some things I've learned after doing some projects in the last four years. In no way are these things guaranteed to be relevant or useful or even correct. But they're food for thought!

But what's the problem?

Define the problem. We hear this all the time but seldom do we really put it into effect, or at least with the required level of sincerity, specificity and clarity. "I want to make a chatting application" might seem self-explanatory as the definition of the required solution, but it's not. The necessity for a clear understanding of the requirements, the scope and priorities, and the assumptions of a solution cannot be emphasized enough. Indeed, this is one of the major reasons why my WiFi Positioning System project was, in subtle terms, a letdown. I vividly remember not being sure about what we were making. To quote the very first line in my logs:

"After days of ambiguity, our group was able to decide firmly about the project idea - Localization using Triangulation."

Except of course, the ambiguity wasn't gone, and to the keen reader, it must be apparent that the topic mentioned in this excerpt is not what we ended up doing.

Is that what we had decided?

It's important to get most requirements or goals clear and sorted out in the beginning. I say "most" because getting the requirements perfect the first time is extremely difficult, if at all possible. If there are stakeholders of the project, it will indubitably prove difficult to solicit and analyze specific and clear requirements from them at the very beginning of the project - one of the biggest drawbacks of the Waterfall model of software development. Invariably, some form of an iterative process involving prototyping and feedback is part of the trade. But the sooner these requirements get clear, the sooner a project will reach completion, and lesser time will be spent pursuing features or improvements that are not required.

It's common knowledge that brainstorming in a group can help a lot with this part, because more aspects of the problem and solution will be explored, uncovering more and more questions that will hopefully be answered later on. What's not common knowledge is that all these (likely diverging) lines of thought should be combined to a single, coherent one early enough, otherwise the benefits of collaboration may turn futile.

Yes, we can!

Can we though?

Another important aspect of planning is feasibility. Gauging the feasibility of a solution is, admittedly, difficult. It requires experience, which I definitely didn't have then, and no one can have enough. However, research can help here. If it's a research project or maybe a data science project, a literature review is helpful, and in most cases, expected. Published work like papers, blogs, source code repositories can help one understand the work that others pursuing similar goals have done, as well as an understanding of the technical difficulty, skills and resources required, and therefore, the feasibility of the project. Did we do a literature review for WiFi Positioning? Yes! But it was very, very haphazard and half-hearted. To make a literature review worthwhile, document it while doing it (and not when you're typing out the final report). Not only does this make researching an active task, but it helps down the line. What's better, going through a 20-page research paper again when you need to, or going through a summary that you have written yourself?

Divide and Conquer

Okay, very trite. But it's only logical that you break down the project into large functional pieces, then break those pieces down further, and refine these pieces until they're granular enough for someone to work on. But if you're working in a team, it's equally important to break down development into pieces that can be worked on independently, in parallel. How often have you worked on something where you had to wait for your friend to push that commit so that you can continue your work? This is not only a waste of time but a sign of an ineffective breakdown of tasks. With experience, this becomes easier and natural to prevent.

With great division of work, comes the great requirement of reproducibility. Hopefully, you'll use a version control system, like git. For those who don't know, version control is the practice of tracking and managing changes to software code. Also use virtual environments and containers wherever appropriate. The focus of team members should be on the task at hand, not on things such as incompatibilities and installations. Use files for configuration (and check the non-confidential parts into version control) instead of typing out those command-line arguments (or worse, redefining constants in source code; been there, done that) every time. Additionally, combine work nicely. That means using a standard workflow to introduce changes to the project, like using pull requests, running tests, and other principles of Continuous Integration.

Write it down, write it down, write it down

Again, it's a no-brainer that you should document stuff. Just like you shouldn't stay up too late and get up early. And just like waking up early, very few take this seriously.

So, one should document stuff? What a drag… Why? We're probably going to write up a final report anyway!

The point of documenting stuff (and humans writing anything in general) is so that you or anyone else can come back to it later if needed. It's communication: between you, your teammates and the same people in the future. It's important to document technicalities and design decisions since they will be revisited. It is important to document tasks, activities and progress so that as a team (or even individually), you know where the project is headed.

If your project involves running many experiments with different conditions or parameters, consider using a dedicated tool to track them (instead of random TXT files, or notes on your phone, or the worst: messages in your group chat). This can be as simple as using a spreadsheet, but if the experiments are more complex, use specialized tools, such as MLflow for machine learning models. The point here is to document and facilitate easy review, search and filtering of all the experiments you do.

A pro tip is to document issues that you have faced, and how you solved them (this is literally how Stack Overflow works; does it sound useful now?). This habit is like an investment. Spend some time now, dig into the problem, and document it. Next time, you're just a Ctrl+F away from solving that problem. This habit also pushes you to actually understand what the problem is and how to solve it, instead of copying something from the documentation or changing parameters randomly (duct tape solutions). To quote my logs:

"In the morning, I was able to solve the GPIO errors. But I still don't know how."

Success: 0 out of 0 tests passed

Iterative and collaborative development needs automated testing, possibly on different levels: unit, integration and end-to-end. Let's face it, regressions suck. They're a headache and a very annoying side-effect of iteration. Tests can seem laborious, but for a large or long-term project, they're worth it, and provide reliability and confidence in code. And they're likely to be mandatory if you're working for an organization. Also, consider using a code quality checker.

Along with tests, technical reviews are of enormous benefit. They fuel discussion and brainstorming, and things like bad design or code smells can be detected. Reviews have a great side-effect: knowledge sharing. We hardly did this for our WiFi Positioning project, and it ended up with team members oblivious of how parts developed by others were working. Indeed, grave bugs and opportunities for improvement stay undetected to date.

Meetings, meetings everywhere

Here's an example of how a college project group generally gets a meeting scheduled: "Let's meet for project TVA at 10 am". That's usually it, followed by a series of messages negotiating the time of the meeting. This is very likely to be an ineffective, diverging and chaotic meeting. Meetings should be limited:

In terms of agenda. There are different kinds of meetings, for example, stand-up meetings for daily updates or task-specific meetings, e.g., discussion on requirements, or an API design, or design considerations of an ML system. A meeting without a preset agenda will likely be an unfruitful one. Since the agenda is undecided, everyone might not be prepared with all areas of discussion
In terms of duration: long meetings can reduce their effectiveness, and in some cases also end up wasting time and proxying for productivity rather than adding value
In terms of frequency; although regularity must be ensured

Connecting People

Constant collaboration or being in a state of perpetual connectivity is the trend, using an array of tools like email, Discord, Slack et cetera. But this habit of staying online and open to communication all the time can be counterproductive. One major reason is attention fragmentation. Constant notifications disrupting your work and intermittent switches to replying to a message break concentration on the task at hand, leading to lower quality work than you would be doing in a state of focus.

According to Cal Newport in his book Deep Work, at work, it's common for people to develop the habit of running their day from their inbox: doing tasks as and when you receive emails about it, and using up much of their time replying to emails rather than actually working. This also prevents them from planning activities beforehand and taking a more proactive approach, since it's easier to use the inbox as a to-do list.

For a project amongst friends, being connected to them constantly for work can easily turn into a hangout. I've lost count of how many times my friends and I have switched from working on a project to playing Skribble on virtual meets.

All these reasons further justify the constraints that I mentioned that you should keep on meetings. Outside of these meetings, try to hold sessions where you actually work in a focused manner.

Finally, always remember, software development is hard. Don't beat yourself up if you find things tough, it's part of the journey. And hey, some pages of that book are still blank!

A Guide to Install TeX on Linux

Akshat Shah — Fri, 10 Dec 2021 17:33:28 +0000

This article was originally published on my website

The Story

Recently, I wanted to install a TeX system on my Ubuntu 20.04 PC. I tried installing MiKTeX after someone recommended it to me (thrice...), but I found myself unable to update TeX packages because of a cryptic SSL peer certificate error. I reported this in an issue to the MiKTeX repo, but was instead told that my system did not support MiKTeX. Great.

Next, I tried to install TeX Live. It seemed simple enough, with a sudo apt install texlive. Can't get any easier than this, right?

So it turns out that the apt package for TeX Live is still the 2019 version. Now unfortunately, this meant that I again could not update packages, since tlmgr, TeX Live's package manager could not verify the signature for a repository. I tried following this post on TeX Stack Exchange, but this didn't work out.

So I decided to ditch APT, and tried following the TeX Live Quick Install docs (which is probably I should have done first). But even in this method, the default mirror which was chosen was too slow, and for some reason, the default, recommended scheme to install TeX Live (called full scheme) installs around 5GB of TeX software - fonts, packages, et cetera. For someone beginning with TeX, this did not sit well with me.

Enough with this rant, let me list down the steps that I followed to install TeX Live on my system. Note that there are most likely better methods/choices that one could use, but these steps worked for me.

What is TeX?

Simply said, TeX is a popular typesetting system, especially in academia, that enables one to write documents that involve a lot of formatting, mathematical equations, tables et cetera. It was made by ACM Turing Awardee Donald Knuth (who is a legend, by the way).

TeX Live Installation

TeX Live is a software distribution for TeX, and includes some programs, macros, packages and fonts. It has a package manager specifically for TeX packages, called tlmgr, that is used to install and update TeX packages.

Download install-tl (the latest one, yay!) using this link. In case this gets changed, you can find the link in the docs (Do read the docs once before installing). Extract the contents from the .tar.gz file. Here's how the contents look:

Run the install-tl Perl script as root. Note that the script does not require root permissions, but requires write access to the directory where it's going to install, which by default is /usr/local. You can change the directory by looking at the prompts of the script.

Note that you can run the installer in GUI mode by the -gui flag (which I would recommend if you're doing this for the first time, but it requires Tcl to be installed).

The --repository option specifies an explicit mirror to use. Try specifying some other mirrors if the default mirror seems too slow. In my case, the mirror provided by NUS (and I stay in Mumbai) was much better than the default one.
```
sudo perl install-tl [-gui] [--repository <URL>]
```
I came across the following error when I tried using an explicit mirror:
```
install-tl: TLPDB::from_file could not initialize from: https://mirrors.dotsrc.org/ctan/tlpkg/texlive.tlpdb
install-tl: Maybe the repository setting should be changed.
install-tl: More info: https://tug.org/texlive/acquire.html
```
Try appending systems/texlive/tlnet to the mirror URL when running the install-tl script. For example,
```
sudo perl install-tl --repository https://download.nus.edu.sg/mirror/ctan/systems/texlive/tlnet
```
After a few seconds, something similar to the following will be visible:

If you scroll up, you can see the scheme selected by default for installation:
<S> set installation scheme: scheme-full. Below that, you can see the number of collections and the space they'll take:

 <C> set installation collections:
     40 collections out of 41, disk space required: 7143 MB

7GB by default, not cool.

Enter "S" as the command (upper or lower case), and then you can select the scheme you want. I selected the "medium scheme", which takes around 1.7 GB, obviously at the cost of lesser functionality. But you can install packages as you like later on, using tlmgr.

After you've selected the scheme you want, enter "R" to return to the main menu, as the prompt will tell you. Then enter "I" to begin installation.

Depending (a lot!) on the mirror you chose, your installation of TeX Live should finally be complete. For me, it took around 15 minutes.

Next, you need to update your PATH environment variable so that it knows where to find TeX Live binaries. If you went along with the default settings, the path you need to add to your PATH would be something like /usr/local/texlive/2021/bin/x86_64-linux. Move around inside /usr/local/texlive to verify this.

You can update your .profile (log out and log in again for changes to take place) or .bashrc or .zshrc or whatever (source the configuration file for changes to take place) by adding this line:

PATH=/usr/local/texlive/2021/bin/x86_64-linux:$PATH

Installing a LaTeX editor

I installed TeXstudio by simply (painlessly) using APT.

sudo apt install texstudio

I'm happy to say that this worked fine! Most TeX editors should be able to configure the backend (TeX Live, in our case) automatically, provided you have set your PATH variable and your shell knows about it.

References

Quick Install - TeX Live

Installing TeX Live over the Internet

For the tlpdb error: An answer in a forum discussing "Problems installing TeXlive in CentOS7"

Coding Problem: Two Sum

Akshat Shah — Fri, 10 Dec 2021 17:20:41 +0000

This article was originally posted on my website.

DailyCodingProblem is a great website that sends coding problems to your inbox daily.

The Question

Given a list of numbers and a number k, return whether any two numbers from the list add up to k.

Examples

[1, 2, 3], k = 4

Yes, since 1+3 = 4

[10, 15, 3, 7], k=17

Yes, since 10+7 = 17

[5, 4, 7, 12, 1], k = 2

Solutions

Brute force

A simple solution would be to iterate over all possible pairs in the array and checking if a pair adds up to k.

for i from 0 to n-1:
  for j from i+1 to n-1:
    if a[i] + a[j] == k:
      return true
return false

This will involve n(n+1)/2 steps, so the time complexity will be O(n^2), and O(1) space complexity.

Sorting and two-pointer approach

We can sort the array and use two pointers, front (starting from 0) and end (starting from n-1):

If the elements pointed to currently sum up to k, then return true
If the sum is less than k, increment the front pointer, since we need to increase the sum and the array is sorted.
Symmetrically, if the sum is more than k, decrement the end pointer. We repeat this until front and end cross each other.

sort(a)
front = 0, end = n-1
while front < end:
  if a[front] + a[end] == k:
    return true
  else if a[front] + a[end] < k:
    front = front + 1
  else
    end = end - 1
return false

Since we are sorting an array (O(nlogn)) and looping with the front and end pointers (O(n)), the overall time complexity of this solution is (O(nlogn)). The space complexity is (O(1)).

You can find the implementation of this solution here

If we are required to return the indices, the sorting approach cannot be used directly. We will have to make an array of pairs of (val, index) and then sort this array.

Using a set or map

We can iterate through the array and keep adding the elements to a set (or hashset, map or hashmap), and for every element, check if the set contains (k-a[i]). If we find this condition to be true, then there exists a pair that adds up to k: the current a[i] and the entry in the set, k-a[i].

set m
for i from 0 to n-1:
  if m.has(k-a[i]):
    return true
  m.add(a[i])
return false

We are making a single pass through the array, finding whether a number exists in the set, and adding an element to the set. If we are using a BST implementation of set (C++ set or map) then the time complexity will be O(nlogn), since every insert/find operation takes O(logn) time. If we use a hashset or hashmap (C++ unordered_set or unordered_map) then the time complexity will be O(n), since insert/find operation can be done in constant time.

You can find the implementation of this solution here

If we are required to return indices of these elements, then need to use a map or hashmap, with the key-value pairs as (value, index).