DEV Community: Santosh Hari

Which Load Balancing Option(s) In Azure Should I Use?

Santosh Hari — Fri, 23 Jun 2023 02:39:47 +0000

Azure docs have a great page on the various load balancing options in Azure that even has an awesome flowchart summing up the choices. However, not being from a networking background, combined with Microsoft’s “special” naming, combined with some sort of memory issue recalling these names from memory meant that even if I had to rely on rote memory when in conversations with customers, I would often mix up the names. For instance, confuse traffic manager and load balancer. So, I decided to understand some of the basics behind cloud load balancers to help become a more interesting conversationalist in this topic: “well actually, you should be using an app gateway there, John”.

You have 4 major options for load balancer in Azure (if you’re reading this in the year 2136, please exit now because this post is outdated af):

Some important concepts:

What is Cloud Load Balancing?

Load Balancing in the cloud is the process of distributing traffic and requests to multiple backend servers. The Load Balancer which does this activity sits between the Internet and cloud servers, listens to incoming requests and can distribute them on a variety of criteria including server load, geography, url route and so on.

Level 4 versus Level 7 Load Balancing

When reading about Load Balancers you will see references to Level 4 and Level 7. These are based on the OSI Model where Level 4 (L4) represents the transport layer and Level 7 (L7) represents the application layer. But what does this mean for us?

Well, L4 load balancers being able to operate at a transport layer means they will be able to handle L4 protocols like TCP and UDP. L7 being the application layer is focused on protocols built on top of these protocols (for instance: HTTP, SMTP). And since they’re at an application level, L7 load balancers can also handle “fancy” operations like security, routing, performance, and so on.

Front Door	Traffic Manager	Application Gateway	Load Balancer
L7	n/a (DNS-based routing service)	L7	L4

Application Delivery – Controllers, Network and such

In the world of networking, application delivery is the process of making an online application (global, regional, local) highly available, secure and reliable. \

Load Balancing in the cloud is not just about distributing packets but also dealing with other issues like handling high transactions with minimal delay, enabling high availability and fault tolerance, and ensuring security. Application Delivery Controller (ADC) is a load balancer that performs a subset or all of these and more. ADCs can be hardware-based (F5 Big IP, Citrix Netscaler) and/or software-based (NGINX Plus, Azure Application Gateway). Based on the features, it stands to reason that ADCs are L7 load balancers.

While ADCs are capable of handling load balancing at a datacenter/regional level, global distribution requires an additional layer of complexity. Enter Application Delivery Network (ADN). An ADN is comprised of a WAN optimization controller (WOC) and one or more ADCs.

Front Door	Traffic Manager	Application Gateway	Load Balancer
ADN	ADN	ADC	n/a

Global versus regional:

Global: indicates presence across regions and, even, on-prem.

Regional: resources usually within VNETs and restricted to zonal/zone-redundant

Front Door	Traffic Manager	Application Gateway	Load Balancer
Global	Global	Regional	Regional (see note below)

Note: There is a service in preview Cross-Region Load Balancer that uses a global frontend public IP address that is advertised across most Azure regions, and a backend pool that contains regional load balancers

Protocols

Protocol	Front Door	Traffic Manager	Application Gateway	Load Balancer
HTTPS	Supported	Supported	Supported	Supported
HTTP	Supported	Supported	Supported	Supported
HTTP/2	Supports HTTP/2 but backend is HTTP/1.1 only	No	Supports Websocket and HTTP/2	Supports gRPC and HTTP/2
TCP	80 and 443 only	All ports	80 and 443 only	All ports
UDP	Not supported	Not supported	Not supported	Supported

Web Application Firewall (WAF)

SQL injection and cross-side scripting type vulnerabilities are the most exploited ones even in “whenever you’re reading this in the future”. WAF is a security service that protects web applications from these common exploits and vulnerabilities, such as SQL injection and cross-site scripting. It inspects (L7) the incoming traffic and can block or allow requests based on predefined or custom rules. Front Door and Application Gateway can function as WAFs

Front Door	Traffic Manager	Application Gateway	Load Balancer
Yes	No	Yes	No

SSL Termination

SSL/TLS termination is the process of decrypting the encrypted HTTPS traffic at the load balancer level, before sending it to the backend servers. This reduces the CPU load on the backend servers and improves performance, which is useful in high volume processing scenarios. Since Front Door and Application Gateway are L7, they can perform SSL termination.

Front Door	Traffic Manager	Application Gateway	Load Balancer
Yes	No	Yes	No

Path Based Routing

If my application had /images or /videos in the url, I want to route the request to the appropriate services/servers. Again these are L7 operations, hence supported by Front Door and Application Gateway.

Front Door	Traffic Manager	Application Gateway	Load Balancer
Yes	No	Yes	No

Caching and Acceleration

Again this is an L7 operation but since caching depends on geography, only Front Door can achieve this

Front Door	Traffic Manager	Application Gateway	Load Balancer
Yes	No	No	No

Health Probes

This indicates routing based on the health and availability of backend servers and all of the above services are capable of this.

Front Door	Traffic Manager	Application Gateway	Load Balancer
Yes	Yes	Yes	Yes

Session Affinity

Once a connection is established, it’s more cost effective to keep it open for an extended duration. This can be achieved with usage of cookies, IP addresses and other techniques. Traffic Manager can achieve this by using IP addresses of the client, and Front and Application Gateway use user sessions.

Front Door	Traffic Manager	Application Gateway	Load Balancer
Yes	Yes	Yes	No

Useful Rules of Thumb

For PaaS web applications, Front Door is probably the best (and often the only) load balancing option.
For global non-HTTP(s) like IoT, start by considering Traffic Manager.
Front Door and Traffic Manager are mutually exclusive and will never (I’m sure some “well actually” will chime in here) be used together.
Application Gateways are the best fit for AKS due to Application Gateway Ingress Controller (AGIC), a service that acts as a monitor for services in the AKS cluster and updates the Application Gateway when it detects changes.
Most global architectures end up putting a Load Balancer or Application Gateway behind a Front Door or Traffic Manager.
Get comfortable mixing and matching options to best suit your need with rule 3 in mind.
Traffic Manager and Load Balancer can handle HTTPs traffic, they’re simply ot the best fit.
Cost: Front Door >> Application Gateway >>>> Traffic Manager >>> Load Balancer

Official Azure Flowchart of selecting load balancing services

Azure Guidance on load balancing options

References

L4 vs L7 load balancing

OSI Layers

On Real Life Lessons We Can Learn From Asymmetric Warfare

Santosh Hari — Wed, 14 Jun 2023 01:03:14 +0000

Real life war strategies are very interesting to me, compared to randos sharing top 10 tips and tricks on LinkedIn. This is a story of a prescient book, how a major warning was ignored and almost turned the tide of World War One. In 1914, right before the war, Arthur Conan Doyle, he of Sherlock Holmes fame, was so worried about the Germans’ submarine superiority that he wrote a freaking novel about it.

Rendering of the real life sinking of RMS Lisutania by a U-boat in 1915

In Doyle’s short story Danger!, Norland a tiny country in the North Sea, gets into a war against Great Britain. The story is about how the tiny nation brings the mighty British empire to its knees with 8 submarines (U-boats)!!! The strategy was simple – the isle of Great Britain was a net importer of food and other vital supplies since the island nation could not sustain enough agriculture and other activities to support its entire population and wartime efforts.

In Danger!, Norland U-boat armada Captain John Sirius (in hindsight) correctly guesses that if you starved the British isles by cutting off the shipping lanes which imported food and other essential supplies, they would have no option but to surrender. In the story, Capt Sirius essentially skips all open confrontation with the might British navy and their destroyers (hilariously named Dreadnoughts) and wages guerilla warfare at sea by concentrating on hapless merchant and supply ships with 8 U-boats only. What made this tactic unique was that marine warfare norms at that point in history (both in real life and in the book) frowned upon attacking civilian and merchant ships.

The aftermath of this successful strategy is described as follows in the book:

In the great towns starving crowds clamoured for bread before the municipal offices, and public officials everywhere were attacked and often murdered by frantic mobs, composed largely of desperate women who had seen their infants perish before their eyes. In the country, roots, bark, and weeds of every sort were used as food. In London the private mansions of Ministers were guarded by strong pickets of soldiers, while a battalion of Guards was camped permanently round the Houses of Parliament. The lives of the Prime Minister and of the Foreign Secretary were continually threatened and occasionally attempted. Yet the Government had entered upon the war with the full assent of every party in the State.

Doyle’s warning was largely ignored by the British admiralty because they believed that no nation would violate the current rules of war. Sure enough, a few months after the land war in Europe, now known as World War 1 broke out, Germany went on the offensive with their U-boats and soon pivoted to include non-military ships. The British navy had no real answer to these attacks until much later in the war and at certain points risked a plight similar to the one in the book.

The other prescient aspect of Doyle’s fiction was the repeated mention of a channel tunnel which would’ve established a land based route between mainland Europe and the isle to have a contingency plan. The Channel Tunnel would not open for another 80 years in real life.

Some real life lessons to learn from Danger! and Norland’s strategy, when going against a superior opponent:

When forced to go against an insurmountable force, avoid direct attacks – Capt Sirius avoided any direct confrontations especially against destroyers and other war ships.
Avoid playing by the rules and conventions largely laid out by the enemy – It was accepted convention that civilian and merchant vessels were to be left alone. Capt Sirius knew his tiny nation stood no chance if he played by convention and directly went against it.
Go for the choke points and focus on destroying them swiftly – Recognizing that the imported food supply was Great Britain’s weak point, Capt Sirius wages all-out war focused against the supply lines only.
Expend only the bare minimum of resources – Capt Sirius leaves the rest of the conventional navy at home, and uses 8 swift moving and strategically placed U-boats to maximum effect.

Conversely, if you’re the larger force:

Learn to recognize your weaknesses and threats to them, however remote those threats may seem
Work on contingency plans

Some interesting reads:

https://www.arthur-conan-doyle.com/index.php?title=Danger!

https://www.bbc.com/news/magazine-28954510

Dead Wake

On Asymmetric Warfare

Santosh Hari — Mon, 12 Jun 2023 00:30:47 +0000

Rendering of the real life sinking of RMS Lisutania by a U-boat in 1915

The aftermath of this successful strategy is described as follows in the book:

In the great towns starving crowds clamoured for bread before the municipal offices, and public officials everywhere were attacked and often murdered by frantic mobs, composed largely of desperate women who had seen their infants perish before their eyes. In the country, roots, bark, and weeds of every sort were used as food. In London the private mansions of Ministers were guarded by strong pickets of soldiers, while a battalion of Guards was camped permanently round the Houses of Parliament. The lives of the Prime Minister and of the Foreign Secretary were continually threatened and occasionally attempted. Yet the Government had entered upon the war with the full assent of every party in the State.

Some real life lessons to learn from Danger! and Norland’s strategy, when going against a superior opponent:

When forced to go against an insurmountable force, avoid direct attacks – Capt Sirius avoided any direct confrontations especially against destroyers and other war ships.
Avoid playing by the rules and conventions largely laid out by the enemy – It was accepted convention that civilian and merchant vessels were to be left alone. Capt Sirius knew his tiny nation stood no chance if he played by convention and directly went against it.
Go for the choke points and focus on destroying them swiftly – Recognizing that the imported food supply was Great Britain’s weak point, Capt Sirius wages all-out war focused against the supply lines only.
Expend only the bare minimum of resources – Capt Sirius leaves the rest of the conventional navy at home, and uses 8 swift moving and strategically placed U-boats to maximum effect.

Conversely, if you’re the larger force:

Learn to recognize your weaknesses and threats to them, however remote those threats may seem
Work on contingency plans

Some interesting reads:

https://www.arthur-conan-doyle.com/index.php?title=Danger!

https://www.bbc.com/news/magazine-28954510

Dead Wake

Connectors, mavens, and salespeople for software developers

Santosh Hari — Thu, 16 Feb 2023 12:00:00 +0000

In America, there is an overwhelming focus by the media on national politics which leaves people jaded to the concept of politics. I would argue that you should still be involved in local/state/national politics but that is not what this post is about. As a result of this association, politics is often a dirty word in the software development world and most software developers actively seek to avoid it. This post is about work politics and that is something that software developers should not only be cognizant about but also actively involved in.

However, being involved in politics does not solely mean going full "et tu, Brute?" on your colleagues. It is also the art of cultivating relationships with people at work and, maybe, over the course of multiple weeks/months/years, serendipitously leveraging that relationship to have a positive impact on the team, company and, of course, thy own self.

Dwight Schrute on workplace politics

Software development is not just about writing code, it’s about telling the computer exactly what you want it do and then making sure that the output of this effort works for people in real life. From my experience in startups, consulting and big tech, the people part is the toughest nut to crack for software developers. But the advice and training given to software developers is also to blame for this situation. Throwing terms like "build trust" and "cultivate relationships" is pretty much useless without context.

If you’re part of a software development team, and, are generally nice and helpful to people that is a great start. But this niceness and helpfulness could also mean you get leveraged by other people when it’s advantageous to them and not reaping the benefits (source: my personal experiences as a nice and helpful guy). Also what about working with people outside the team and with customers?

This is where it helps to identify and classify people, particularly ones that can help boost the impact of your work. I first read about connectors, mavens and salesmen in Malcolm Gladwell’s Tipping Point. My distaste for Malcolm Gladwell had grown over time because of his love for using anecdotes as evidence but his classification of people is still on point. Identifying the connectors, mavens and salespeople is key to doing politics at work.

Connectors spark the intro

Connectors are people, as the word suggests, who are ultra-connected within an organization. An easy way to identify this type of person is to seek out someone who often uses terms like "to solve X problem you should probably talk to Y person". A connector who volunteers this information freely is often the helpful kind and someone who you can go to with general problems. Once you cultivate a relationship with a connector, remember that you become part of their network and they may direct someone to you. So make sure you do not treat this as a single lane road, if you ask for help, be willing to help.

Mavens are the librarians of your organization

Mavens are people who connect you with information, instead of people. Mavens are the people who often know what was said in the company town hall or know where to find obscure information regarding the organization, product, or project. They will actually read the internal documents and the monthly update emails. A helpful maven again will be seen volunteering this information and is definitely someone you can go to in order to figure out how to obtain information. Again, this is not a "take only" relationship, you should be willing to voluntarily share useful information to a maven like updates in their area of interest that they may not know about yet.

Salespeople use Jedi mind tricks to get everyone to agree to their proposal

Salespeople in an this hierarchical context are persuaders. They have this magical ability to get others to agree with them without resorting to nastiness or pulling rank. An example of a salesperson would be someone who is excitedly talking about new product/technology the team should try and fast forward few weeks and the entire team is using it. What happened was the salesperson used their superpowers/Jedi mind tricks to push their agenda but also made us feel excited about adopting it. This archetype of person is harder to find in a software development setting. So once you find one, put the proverbial ring on it and never let them go. Salespeople do have a reality distortion field around them and as such we should exercise caution to make sure it does not turn into a "give only" type relationship where you’re the one giving all the time.

And finally, remember to both give credit if you benefit from one of these folks and also, that it is always OK to help these people and others in a pinch without expecting anything in return. Go forth and be political at work, o’ developer of the software.

How to verify your GitHub account on Mastadon

Santosh Hari — Wed, 15 Feb 2023 12:33:00 +0000

Mastadon is the social media service of choice that large sections of people, techies in particular, are flocking to after the recent events at Twitter, mainly because it’s an non-algorithmic decentralized platform.

On a related note, you can follow me on Mastadon.

Verifying your GitHub on Mastadon is a 2-step process

Paste your verification link in your GitHub Profile
Paste your GitHub link in your Mastadon Profile Metadata
That’s it!!

Twitter had an opaque process for verification with the previous management, and, now infamous $8 verification for trolls and bots with the current management> Mastadon, OTOH has a self verification process where you can verify yourself by posting a piece of code on a website or service. The instructions for this can be found under the Edit Profile section as shown below.

This will work with websites where you can edit the homepage source. However, the issue with GitHub is different. The most obvious space appears to be the readme.md in the profile page. However, if you edit this page and insert the verification code, it does not work. You can read a detailed explanation on why.

The trick is to take the verification url and plug it into your profile settings directly ( Edit Profile button on homepage) as shown below.

After updating the GitHub profile with verification code, please make sure you add your GitHub link to your Mastadon Profile Metadata as shown below.

Since the tech crowd loves to use GitHub, why not verify on Mastadon with your GitHub profile? Please share these tips with the community.

Office 365 Outlook Automatic Replies – your very own vacation virtual assistant

Santosh Hari — Mon, 16 Aug 2021 10:51:00 +0000

Ever go on vacation, only to be bombarded by work emails and meeting invites? What are you supposed to do? Should you respond to these invites with either

a. "no" if the meeting is during the vacation, or,

b. "yes" if the meeting is after vacation

This means you would have to spend precious vacation time parsing through these invites figuring out if they fall within your vacation window or not. What if you had your own automated virtual assistant to take care of these bureaucratic issues while you’re on a beach sipping mojitos? Isn’t that what computers are supposed to do, instead of taking over the world like Skynet wants to?

You can now let Office 365 Outlook take care of this bureaucratic business while you’re on vacation. Outlook will:

Block your calendar for the duration of the vacation
Automagically decline all invites sent you during your vacation that fall within your vacation window
Decline any meetings you might have previously agreed to during this window, like daily standups
Cancel any meetings you previously setup that falls within this window.

Please note: as of publishing this post, this feature is only available on the web version of Office 365 Outlook https://outlook.office.com

Screenshots below show how to setup this up. Enjoy your vacay!

Using app secrets in #dotnetcore console applications

Santosh Hari — Fri, 30 Jul 2021 08:01:00 +0000

I was writing a sample dotnetcore console application for a talk because why not! I felt using a sample aspnet core web app was overkill. The app was connecting to a bunch of Azure cloud and 3rd party services (think Twilio API for SMS or LaunchDarkly API for Feature Flags) and I had to deal with connection strings.

Now I have a nasty habit of "accidentally" checking in connection string and secrets into public GitHub repositories, so I wanted to do this right from the get go.

I started with this official documentation on adding configuration in a new .NET console application. To start with, add a package reference to Microsoft.Extensions.Hosting (example with dotnet cli)

dotnet add package Microsoft.Extensions.Hosting

Next, modify Program.cs to instantiate a new instance of the HostBuilder class with pre-configured defaults

using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;

namespace Console.Example
{
    class Program
    {
        static async Task Main(string[] args)
        {
            using IHost host = CreateHostBuilder(args).Build();

            // Application code should start here.

            await host.RunAsync();
        }

        static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args);
    }
}

According to the Microsoft docs, The Host.CreateDefaultBuilder(String[]) method provides default configuration for the app in the following order:

ChainedConfigurationProvider : Adds an existing IConfiguration as a source.
appsettings.json using the JSON configuration provider.
appsettings.Environment.json using the JSON configuration provider. For example, appsettings.Production.json and appsettings.Development.json.
App secrets when the app runs in the Development environment.
Environment variables using the Environment Variables configuration provider.
Command-line arguments using the Command-line configuration provider.

While options 1, 2, 3, 5 and 6 sound like they would work, I would really like to use option 4 since it explicitly talks about app secrets and I believe secrets should be stored separate from config. Additionally, I really liked the app secrets experience, including support for POCO, when I worked with aspnetcore web applications. So, I decided to go with option 4.

But the Microsoft docs takes you a page that shows you Safe storage of app secrets in development in ASP.NET Core. This might work but I would like to keep my application as a simple console app not a web app. Still there are some instructions that would be useful on here.

The app secret values will be stored in a JSON file in the local machine’s user profile folder. For Windows this path will be %APPDATA%\Microsoft\UserSecrets\secrets.json and for Linux or MacOS this path will be ~/.microsoft/usersecrets//secrets.json.

Use the Secrets Manager command line to enable secrets for the project

dotnet user-secrets init

This puts a guid in your .csproj file. If you take this guid and plug it into the paths above instead of , you will get access to the the secrets.json file that will store the secrets for the application, in case you want to review them during debugging. Please note, the init function will not create the file. The file will be created and modified as you continue to add and modify secrets.

Next, we come up with POCO classes that can be used to map to the secrets. In this case, I split MyAppSecrets into AzureSecrets and ExternalSecrets and have dedicated sub-classes for each.

    class MyAppSecrets{
        public AzureSecrets CloudSecrets{get;set;}
        public ExternalSecrets UtilitySecrets{get;set;}
    }
    class AzureSecrets{
        public string SQLConnectionString{get;set;}
        public string CosmosConnectionString{get;set;}
    }

    class ExternalSecrets{
        public string TwilioApiKey{get;set;}
        public string LaunchDarklyApiKey{get;set;}
    }

Now to add the secrets themselves. The dotnet user secrets tool does not store nested properties as proper json. Instead it uses a : to separate the properties structure. Let’s take the above POCO classes, to access SQLConnectionString, we would use TopLevelClassName:PropertyClassName:PropertyName as the secret name, for instance, MyAppSecrets:CloudSecrets:SQLConnectionString. To set a secret, you would use a dotnet cli command as below

dotnet user-secrets set "MyAppSecrets:CloudSecrets:SQLConnectionString" "sqlconnstring"

Repeating this for all the sub-classes and properties you could end up with a secrets.json file that looks something like below – not very readable, I know

{
  "MyAppSecrets:UtilitySecrets:TwilioApiKey": "twilioconnstring",
  "MyAppSecrets:CloudSecrets:SQLConnectionString": "sqlconnstring",
  "MyAppSecrets:CloudSecrets:CosmosConnectionString": "cosmosdbconnstring",
  "MyAppSecrets:UtilitySecrets:LaunchDarklyApiKey": "launchdarklyconnstring"
}

Now that we have the secrets stored and the POCO class to retrieve it, app secrets should be enabled when we run the app in development environment. It turns out you can tell the HostBuilder to use a particular environment right after instantiating it.

Host.CreateDefaultBuilder(args).UseEnvironment("development");

Convenient, isn’t it? You can add and delete the UseEnvironment("development") code snippet and witness for yourself how Secrets are added and removed from the list of configuration sources by debug watching the sources variable in below code.

        static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args).UseEnvironment("development")
            .ConfigureAppConfiguration((hostingContext, configuration) => {
                var sources = configuration.Sources;
            });

But I really don’t want to have to deal with all these source since I’m only interested in the App Secrets source. I can do this using a configuration.Sources.Clear() call from the CreateHostBuilder method. Having done that I can build a configuration source for my app secrets and read it on to a static variable to use in the program as follow

Add a static variable for the Program class

static IConfiguration Configuration;

Then modify your HostBuilder method to build an app secrets configuration.

        static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args).UseEnvironment("development")
            .ConfigureAppConfiguration((hostingContext, configuration) => {
                configuration.Sources.Clear();
                Configuration = configuration.AddUserSecrets<MyAppSecrets>().Build(); 
            });

And boom goes the dynamite. Now you can use the static variable in your Main class to either access your secrets as a key value pair or by mapping it to a strongly-typed class as shown below

static async Task Main(string[] args)
{
    using IHost host = CreateHostBuilder(args).Build();
    //mapping static variable to strongly typed class
    var appSecrets = Configuration.GetSection(nameof(MyAppSecrets)).Get<MyAppSecrets>();
    Console.WriteLine($"Strongly typed mapping {appSecrets.CloudSecrets.SQLConnectionString}");
    //access secrets using key value pair
    Console.WriteLine($"Key value pair {Configuration["MyAppSecrets:CloudSecrets:SQLConnectionString"]}");
    await host.RunAsync();
}

But wait! There’s more. If you don’t want to have to deal with using the dotnetcore cli to enter awkwardly structured secrets like MyAppSecrets:CloudSecrets:SQLConnectionString, you can modify the secrets.json file directly with properly formatted JSON value and your strongly type mapping will ensure that these values are available to you either as strongly typed class properties or config key value pairs

{
    "MyAppSecrets":{
      "UtilitySecrets":{
          "LaunchDarklyApiKey": "launchdarklyconnstring",
          "TwilioApiKey": "twilioconnstring"
      },
      "CloudSecrets":{
          "SQLConnectionString": "sqlconnstring",
          "CosmosConnectionString": "cosmosdbconnstring"
      }
    }
}

Feel free to peruse the easy to read source code. I will add more instructions in the GitHub repo but this blog should suffice for now.

OKRs – In Practice

Santosh Hari — Wed, 03 Mar 2021 13:00:00 +0000

The approach in John Doerr’s book: Measure What Matters: How Google, Bono, and the Gates Foundation Rock the World with OKRs is based off Google’s OKR Playbook. While it’s good to know the practical aspects, keep in mind a good portion of the company wide practices probably only works for Google but there are some useful takeaways.

Salient features of Google’s OKRs

Think big, not incremental. Don’t expect to hit them all
Grading on a 0-1 color scale

Points	Color
0.0 – 0.3	Red
0.4 – 0.6	Yellow
0.7 – 1.0	Green

Google’s OKR Rating Technique

Writing effective OKRs

Objectives == What

express goals and intents
aggressive but realistic
achievement should be no-brainer for a rational observer

Key Results == How

milestones adding up to achievement of objective
outcomes not activities.

Cross-team OKRs

Include all teams involved in project. For instance, if Main website development, central SRE and central networking must deliver to support new SLAs for website, all three teams should have OKRs to deliver their part of the project.

Committed and Aspirational OKRs

Committed	Aspirational
Must deliver clear objectives or business will be impacted	Future facing nebulous path, how we want the world to look
Schedules and resources will be adjusted to achieve	No clear idea on schedules and resources
1.0 score required to succeed. Score of less indicates errors in planning and execution	0.7 score is acceptable with the idea that once teams to get to 70% they will try harder to make 100% for an aspirational objective
Teams expected to escalate if 1.0 score is in danger	Remain on team’s OKR list until completed, sometimes moved to another team with more expertise and/or bandwidth

Committed OKRs v. Aspirational OKRs

OKR-writing mistakes and traps

No clear distinction between committed (keep lights on) and aspirational (this is how the world should look) OKRs. Inverting committed and aspirational OKRs also invites.
Teams writing OKRs without understanding what customers want. The classic mistake is teams write OKRs believing they can achieve certain objectives by simply repeating what previously worked.
Aspirational OKRs are timid, i.e., start from current state and make incremental changes. Instead ask “What could my (or customers’) world look like in several years if freed from most constraints?”.
Team is sandbagging. Committed OKRs should consume most but not all available resource. Committed plus aspirational OKRs should consume all resources and stretch team (sounds like a recipe for burnout).
Fails the “who cares?” test aka Low Value Objectives (LVOs). For instance, “increase build speeds for main website repo by 10% by Jan 1 2022” is a classic LVO. Instead For instance, “increase build speeds for main website repo by 10% by Jan 1 2023 …. resulting in faster feedback to developers increasing productivity by x% enabling developers to decrease bugs by y%” has a clearer economic objective (dev productivity and less bugs in product).
Not writing enough KRs for Os. Scoring 1.0 on all KRs should result in achievement of the O. Classic example would be an objective to improve user experience and the key results would focus on the mobile experiences but not desktop or vice versa.
KRs use team internal language. For instance “Launch Index Engine v4.1”. Instead use “Launch Index Engine v4.1 to improve query performance by 25%”.
Uses business dates instead of real dates, for instance, end of quarter or end of fiscal year dates instead of actual dates for time boxing effort.
Ambiguous metrics. For instance, “support 1 million users”. Is it 1 million users concurrent or per day or per week?
Using same OKRs for entire large groups. Instead use, high level ones for entire group but more details ones for sub-groups. Cross-team OKRs should have KRs in each sub-team.

OKR Concepts

The first 4 are also called the 4 superpowers in the book.

1. Focus and Commit to Priorities

Run short-term (usually committed) and long-term (usually aspirational) OKRs in parallel.
Rollout should start with upper management to make them accountable too. In fact, without organizational buy-in and management conviction, OKRs are bound to fail
OKRs require a shepherd to focus individuals on priorities.
Commit 3 to 5 OKRs per cycle to prevent dilution
Departmental objectives requiring support from outside should become company OKRs
Each objective should be limited to max of 5 measurable, unambiguous, time-bound key results
If one key result should be stretched beyond one cycle, it should go into an objectives that has similar cycles

2. Align and Connect for Teamwork

Everyone from CEO down should make their OKRs transparent. Employees should relate OKRs up to company priorities
Messaging around priorities requires constant repetition
Even if objectives are driven from the top, it’s important to allow flexibility on key results from frontline contributor. Innovation dwells less at a company’s center than it’s edges
Healthy mix of top-down and bottom-up OKRs
Cross-team priorities requires horizontally shared OKRs. This should be explicit and transparent throughout the org
When OKRs are modified or dropped, every stakeholder should be informed

3. Track for Accountability

Continuous assessment and objective grading starting at the top. Important for leaders to take risks, sometimes fail and admit to failures publicly.
Focus on measurement and less on extrinsic rewards for achievement
OKR shepherd should be involved in regular check-ins
Prescribed in the book: weekly 1-on-1 meetings with contributor and managers, monthly departmental meetings
OKRs must evolve as necessary, even mid-cycle. Counterproductive to hold on to outdated or unrealistic goals
Important to self assess and celebrate victories at end of each cycle
Use public, collaborative, real-time goal setting systems (none recommended by book)

4. Stretch for amazing

BHAG == Big Hairy Audacious Goal aka aspirational
Important to separate committed goals and BHAGs
Risks and failures in short cycles should be encouraged for quick feedback
All goals should stretch even if some risk not being met
Replace incremental OKRs with exponential ones. 10x (now toxic) comes from this
Stretch goals could be rolled to next cycle, on a reasonable basis

5. Continuous Performance Management

Move from annual performance management to continuous performance management. This is necessary to monitor and address issues throughout the cycle
Tying goal attainment to bonus checks will cause sandbagging and risk-averse behavior
Important for performance evaluations to be transparent and catering to an individuals strengths and ambition
Motivate intrinsically with purposeful work and opportunities for growth
Implement ongoing CFRs (conversations, feedback and recognition) along with structured goal setting
Contributor should set the agenda for performance conversations. Manager should learn and coach
Performance feedback should be unconstrained by org chart. Leverage anonymous surveys for real-time feedback
Provide opportunities for peer feedback and peer recognition

6. Importance of Culture

Align top-line OKRs with org values
Values should be conveyed by deeds, words are cheap
For collective objectives, assign key results to individuals
Promote opportunities for interpersonal/volunteer support for additional help on actions
Address issues around accountability and trust before rolling out OKRs

Sample OKR cycle

Settings OKRs at company, team and individual levels

4-6 weeks before quarter: Leadership brainstorms top-line OKRs for company.
2 weeks before quarter: Communicate output of step 1 company-wide
Start of quarter: Teams develop own OKRs to align with top-line OKRs
1 week after start of quarter: Individual employees share their own OKRs in 1-on-1 settings with manager to make sure they’re aligned at the team and company level.
Throughout quarter: Individual employees measure and share progress with regular check-ins with managers. This is a time to assess the progress and re-calibrate, if needed.
Before end of quarter: Individual employees score their OKRs, self assess and reflect on accomplishment.

Sample OKRs

Objective Improve web service SLA to 4 nines

Key Result 1 Launch v4.1 web service by March 30 with updates for stability

Key Result 2 100ms response time for web service

Key Result 3 99.99% uptime for web service

Key Result 4 Add regional redundancy for web service

Don’t just say hello in chat

Santosh Hari — Thu, 25 Feb 2021 14:00:00 +0000

So the official Microsoft Twitter account dropped a tweet that deeply resonated with me.

Please don’t just say hello in chat.

— Microsoft (@Microsoft) February 24, 2021

So what’s wrong with just saying hello in chat? For context, this assumes business chat like Teams or Slack. Well, consider the below exchange that happened earlier this week – Monday and Tuesday (aka Yesterday in below screenshot) – between me and a collaborator on a project.

A collaborator (West Coast) messaged me (East Coast) after hours (for me) with a "Hi" and no context. I generally don’t respond to chat messages after hours unless someone tags it "Urgent" or "Important" – as is our standing team policy. Well, as you can gather from above exchange, I did not respond until nearly 15 hours after the initial message was posted and it took another 3 hours to arrive at the context "something was not/working". As we work remote (and global), asynchronous communication is not only important but also necessary. But asynchronous communication only works if we give it context. We’re hardwired to be polite but saying "hi" in chat and then waiting 18 hours to provide context is a recipe for failure when it comes to asynchronous communication. Other forms of saying "hello" in chat:

"Hello X, got a minute?" (Instead: "Hello X, if have a minute can you please take a look at X?")
"Hey! I have a quick question" (Instead: "Hey! followed by actual question)

For more examples, read nohello.com. On a related note, I find people who use nohello as a permanent status on their chat as a little rude. So what is the solution? A passive aggressive chat bot that responds to Hi with a gif? Chime in with thoughts below

On a lighter note, that tweet evoked a hearty agreement from me, which lead to this hilarious exchange culminating in a response from the official Microsoft Twitter account. Always a good day when that happens.

You're welcome.

— Microsoft (@Microsoft) February 25, 2021

Intro to OKRs

Santosh Hari — Mon, 22 Feb 2021 13:00:00 +0000

These are my notes from reading John Doerr’s book: Measure What Matters: How Google, Bono, and the Gates Foundation Rock the World with OKRs.

What are OKRs?

OKRs as defined in the book:

A management methodology that helps to ensure that the company focuses efforts on the same important issues throughout the organization.

John Doerr: Measure What Matters

Who can use OKRs?

While the term OKR, which probably originated at Google, was largely used for organizational goal settings and related actions, this technique can be used by organizations of any size and even individuals. Understanding OKRs and helping implementing them can also prove useful for non-leadership rung employees since this can help measure your impact on the organizational goals.

How to do OKRs?

Objectives = many Key Results
Objective (1)–>(has many) Key Results.
OKRs are a management methodology to ensure focus on same issues company-wide.

Objectives	Key Results
– “WHAT is to be achieved, no more no less”	– “benchmark and monitor HOW we get to the objective”
– significant, concrete, action oriented, and (ideally) inspirational	– specific, time-bound, aggressive yet realistic, measurable and verifiable
– long lived, often year or more, can be rolled over	– quarter or less, evolve as work progresses

Comparing Objectives and Key Results

Objective = sum of all Key Results, if all key results achieved then objective should be complete

What does a Sample OKR look like?

Express a clear objective and follow it up with measurable key results.

Objective Improve web service SLA to 4 nines

Key Result 1 Launch v4.1 web service by March 30 with updates for redundancy and stability

Key Result 2 100ms response time for hosting platform

Key Result 3 99.99% uptime

Key Result 4 Add regional redundancy

Book Summary

The book is divided into 3 parts: superpowers, applications and methodology

Part 1: Superpowers

Focus and commit to priorities: OKRs focus teams and companies on work that’s important and reject superfluous work. This forces leaders to make hard choices on priorities. Removes any lingering confusion on what people are supposed to be working on.
Align and connect: OKRs should be transparent throughout the org. Individuals should link their OKRs up to team and teams should link up to company and sideways to cross-dependencies with other teams.
Track for accountability: OKRs are data driven and enforce periodic check-ins, objective grading and continuous reassessment. The most important part is that any key result that is in danger triggers action to put it on track or be revised/replaced, if necessary.
Stretch for amazing: By focusing, align and tracking, OKRs motivated people and companies to overperform.

Part 2: Applications

CFRs: Conversation, Feedback and Recognition add up to “continuous performance management”, a replacement for annual performance reviews.
Continuous Improvement: Structured goal setting and continuous performance management help organizations on the long road to operational excellence by improving one day at a time.
Importance of culture: OKRs and CFRs aim to define and build a positive culture by aligning teams towards common objectives, creating transparency and accountability, and, structured goal setting.

Part 3: Methodology (to be published in next post)

Notable Quotes from the book

“Ideas are easy. Execution is everything.”
Edwin Locke: “hard goals” drive performance more effectively than easy goals. Second, specific hard goals “produce a higher level of output” than vaguely worded ones.
A two-year Deloitte study found that no single factor has more impact than “clearly defined goals that are written down and shared freely. . . . Goals create alignment, clarity, and job satisfaction.”
Andy Grove “Bad companies, are destroyed by crisis. Good companies survive them. Great companies are improved by them.”
Leaders must get across the why as well as the what. Repeat until the entire team understands the “why”. LinkedIn CEO Jeff Weiner *”When you are tired of saying it, people are starting to hear it.”
Andy Grove “For the feedback to be effective, it must be received very soon after the activity it is measuring occurs. Accordingly, an [OKR] system should set objectives for a relatively short period. For example, if we plan on a yearly basis, the corresponding [OKR] time should be at least as often as quarterly or perhaps even monthly.”

Statutory Warning

Focusing on goals solely leads to tunnel vision, infighting and possible unethical behavior. The book gives the example of Wells Fargo where the number of new accounts opened was set as a goal.

My critiques of the book

Too many feel good survivorship bias stories
Some of the case studies and examples are outdated, for instances, Google no longer does 20% time for side projects
Silicon Valley name dropping is rampant throughout the book
No negative case studies featured. The book appears to make the supposition that either OKRs work or you’re not doing it right
If committed OKRs take up most of your resources and aspirational OKRs which run multiple years are supposed to stretch your team, how is a team not supposed to burnout?

On remote work and time zones

Santosh Hari — Mon, 24 Feb 2020 10:27:01 +0000

Recently, I had the pleasure of working with a client on the opposite coast. However, this presented its own set of challenges, the primary one being dealing with time zones. So, I naturally learned to adjust my communication to factor in time zones since it would be embarrassing for me to schedule a 2pm call and then have the client not show up.

Protip: when working remotely always state timezone clearly. Heck, do this even when not remote

"Are you available at 2pm US Eastern today?"

— Santosh Hari (@_s_hari) February 17, 2020

I enable multiple time zones in Outlook, my chief timekeeper at work, as shown below.

Fwiw, you can also enable multiple timezones on Google Calendar

But the original tweet appeared to have sparked interest in the topic among my followers, largely techies and I received a lot of useful tips.

David Haney uses relative time, which is great for same-day scheduling.

I also use relative time, as it applies the same way to everybody: "are you available in 90 minutes?"

— David Haney (@haneytron) February 17, 2020

Steve Lorello uses UTC, which works great when scheduling meeting with a software-focused crowd.

I usually use UTC when scheduling meetings with folks outside my time zone lol

— Steve Lorello (@slorello) February 18, 2020

Michael Godfrey uses the 24-hour calendar aka military time to avoid confusion.

I have learned to do it all the time. Just to be safe. And to use 24 h time.

— Michael Godfrey (@mgodfre3) February 18, 2020

Oscar “OZ” Zamora always specifies the location. This is something I could start doing myself since I communicate with Canadian coworkers all the time.

And location. Coworker from Mexico extended a teams meeting for today; holiday.

— Oscar "OZ" Zamora (@ZamoraO) February 18, 2020

A friend with a private Twitter account also mentioned blocking off time on the calendar if you don’t need to be disturbed. For instance, if you need to do heads-down coding all afternoon just block the afternoon off. Also, if you have coworkers or clients reaching you after-hours block those off on your calendar.

Do you have any additional time zones and calendar tips? Feel free to share in the space below