DEV Community: Shubheksha Jalan

How I moved from India to Europe for a tech job

Shubheksha Jalan — Fri, 14 Aug 2020 11:08:21 +0000

I moved from India to London for a new job in Jan 2019 and have been living here ever since. I wrote about the experience right after I moved. Since then, I have gotten lots of questions from folks in the same boat about how I managed to pull it off, so I thought it'd be best to compile everything I learnt into a blog post for anyone else looking to relocate for a European job.

Whilst initially I considered moving to the US, I gave up after learning about how hard it is to get permanent residency there as an Indian citizen and focussed on (western) Europe, so this post is heavily biased towards that region.

Moving is hard and stressful as is, but moving to a completely new country is even more daunting. A lot of folks think it's impossible and you need to be a genius to pull it off. I can assure you that that's very much not the case and it's very doable if you have the patience to stick through the process. Let’s dive in!

Finding a job with visa sponsorship takes research and patience

This topic is what I get almost all the questions about. It's the hardest and often the most confusing part as it's not very straightforward to find employers who would sponsor a visa. More so if you're not looking to work at big tech companies. In almost all countries, you need a job if you wish to relocate.

Caveat: your entire life will be tied to your job and if you lose it for whatever reason, you may be forced to leave the country after a short period of time.

A few companies clearly mention up-front whether or not they sponsor visas. Your level can have an impact - some companies prefer to sponsor relatively senior folks than folks early in their career. Remember that visa sponsorship is a costly and cumbersome process for the company to undertake. However, most big enough startups are open to sponsoring folks at all levels. I'm not sure how Covid-19 will impact this in a remote-first world.

I looked for jobs through a variety of sites: LinkedIn, Twitter, various job boards. Going through a mutual is the quickest method by a long measure, but I cold-emailed and got in touch with folks directly too. Some governments, like the UK, also publish their database of employers who hold a valid sponsorship license, which you can sift through. It is often a very time-consuming process and I haven't found an easier way to do it. Based on what I remember, there were significantly more options for frontend positions than backend positions. However, this might have changed now.

Be clear about wanting to relocate from the start

Always clarify before interviewing that you're looking to relocate if you're not clear whether or not your prospective employer is open to sponsoring you. It'll be a huge waste of time for everyone involved if you go through the entire process and find out that they're not willing to sponsor you. Explicitly state what you're looking for in terms of relocation in one of your initial calls rather than waiting.

Having more than one option maximises your chances

Don't put all your eggs in one basket. It's a hectic and lengthy process and you want to have multiple options. I interviewed with 5 companies across 5 different countries in order to make sure I have options in case something goes wrong somewhere.

How was the process for applying for the visa?

Once you've found a job, the next logical step is to apply for a visa.
This varies a lot country-to-country. In the EU (except UK and Ireland), countries have their own work visas as well as something known as a Bluecard. This is an EU-wide work permit with every member country having their own rules around what is permitted and timelines for getting residency. Some employers prefer to apply for a country-specific visa despite having the option to apply for a Bluecard. 🤷‍♀️

The UK and Ireland have their own separate visa systems that only allow you to work in that country.

This will usually be taken care of by your employer. Typically, they’ll employ an immigration law firm that'll take care of drafting and submitting your application, and you'll just need to provide them with the right documents. Timelines vary a lot depending on the country and type of visa and you'll need a lot of patience to get through this process and keep your anxiety at bay.

Clarify who is paying for what

Check with your employer about whether they’re funding your visa as they cost a lot of $$$$$. You don't want to be stuck paying that out of pocket.

Read your contract carefully

Also read your contract to make sure you understand if there are any special terms attached to it like having to pay back some of the visa or relocation costs if you leave before X months/years.

What was the actual move like?

Once you have your visa, you're set to actually make the move. Wooooohoooo, the hard part is over! 🎉 Depending on when you get your visa, you'll negotiate a start date with your employer.

You’ll need some savings to kickstart your life

Have some savings you can rely on before you get your first paycheck and/or relocation bonus. If you don't have enough savings, make sure you clarify that with your employer and ask for your bonus upfront. There's no shame in doing that. 🤗

Ask for a relocation bonus

Negotiate a relocation bonus, especially if you're coming from a country with a significantly weaker currency, like India, or if you’re moving somewhere with a higher cost of living. Your savings will evaporate in front of your eyes before you know it. Usually, you can use it for all sorts of costs associated with moving: shipping your stuff from back home, furnishing your new place, etc.

Your employer will usually pay for your flight to the new country and some kind of temp accommodation till you find somewhere to live. This may or may not come out of your relocation budget.

Don’t relocate on the day you’re starting your new job

Fly in before you're due to start so that you can beat jet lag and familiarize yourself with your new home. I flew to London a couple of days before I was due to start my job because I wanted to make sure I could get used to a completely new country (and city!) for a few days beforehand.

It’s a lot of work and it took me around 4-5 months to get it all sorted from arranging interviews to finally moving. Patience is key but at times it can feel impossible. I want to stress that if you’re willing to put in the work, it’s definitely doable.

I tried to distill most of widely-applicable advice I could think of in this post. If you've more questions, feel free to email or DM me on Twitter and I'll try my best to answer by augmenting this post. 😊

Shout out to Miles and Rika for reviewing my drafts. 💜

Lessons learnt in year three as a software engineer

Shubheksha Jalan — Tue, 04 Aug 2020 14:03:26 +0000

18th July marked 3 years to the day since I started working as a software engineer full time. I published a 2 year retrospective last year, so I wanted build up on that and make it into an annual ritual (hopefully I can stick to it 🤞).

The goal of these posts would be to share what I wish someone would have told me when I first started working. However, I have had to learn over the course of many years, so I hope this will help folks who are new to the industry and looking for some guidance.

Doing isn't enough, you need to know how to market your work

This was my biggest and most hard-hitting realization so far in my career. I've always been the kind of person that sits in the corner, puts their head down and gets stuff done. It took me a while to realize that that's not enough. Knowing how to present, talk about and market your work to the right people is a critical skill. Nobody teaches you how to do this, but trust me, you'll thank yourself further down the line in your career if you invest some time in learning how to do this effectively. It'll make a huge difference in your career trajectory. Learning to shout about your work is hard, I know, but you'll often not reap the rewards if you leave it there.

Titles do matter, even if they'd like you to believe that they don't

There's a ton of discourse about this on tech Twitter and I often see men who are already senior in tech tell folks just entering the industry that they shouldn't run after titles. Honestly, I think this is doing them a disservice. My main gripe with it is that it completely fails to consider the fact that titles decide what rooms you are allowed in. Titles/levels give you legitimacy, especially when you don't enjoy the privilege of being assumed to be competent. It makes people, who otherwise wouldn't, listen to you and take you seriously. They're worth it for proving that you indeed know what you're talking about.

The dangers of blindly climbing a career ladder

Titles/promotions/levels do matter and you shouldn't let them take a back seat in your career. However, you need to establish a balance that works for you. If you spend all your time and energy chasing a promotion, then it starts to almost feel like having a second job. You want to avoid getting into that situation as it can prevent you from focussing on your actual job and drain you of all the excitement and energy. If you find yourself blindly trying to climb a career ladder while your actual job and growth has taken a backseat, it's a red flag, and it might be time to start looking for something new. It's sad that it has to be this way but it's not by chance that the most common and fastest way to get promoted or a raise in this industry is to switch jobs. Protect your energy to invest in better and more fulfilling pursuits.

Sponsors are like cheat codes in the career game

There's a broader point here about investing early on in building a solid network which I touched upon in my post from last year. Here, I want to specifically touch upon the importance of building and investing in longer term relationships. If you want to learn more about the concept of mentors vs sponsors and how they differ, Lara Hogan has an excellent blog post and talk that covers it better than I could here.
Having folks who are trusted by people around you, are willing to stick their neck out for you and help you grow is like having access to cheat codes while playing the career game. What sponsorship has looked like for me in practice:

Shouting about me and my work in rooms I am not in
Trusting me with opportunities that stretch my comfort zone but having my back and making sure I don't spread myself too thin
Being a good sounding board for when stuff gets hard and making your sponsee feel heard and understood. Then actually doing something about it

I can't emphasize the importance of having folks who are eager to open doors for you at every step of the way. It's not just a confidence boost that there's someone out there who wants to lend their privilege and help you build credibility, but it also goes a long way in helping you progress much faster.

Programming gets easier over time

Programming is hard. I know people who've been doing this for a while don't like to talk about that but that doesn't negate it. It took me a while to form mental models of how things work. Now I like to see programming as a way to keep augmenting those mental models, breaking down and forming new ones as we go. Something that has taken me by surprise time again is that you might not see everything coming together but after spending some time, things automatically start to click and those are the moments I live for. It makes sense and becomes a hell of a lot easier after a while, I promise.

I love building things with code. Programming brings me so much joy all these years later. And even though it's not always been easy, I can't really imagine doing anything else with my life. 3 years in -- many more to go. To bigger and better things. ✨

These points could've been self contained blog posts, so if you enjoyed this post and want me to write a post on any or all of the points above, do drop me a line via Twitter/email, I'd love to hear from you. 💜

Retries in distributed systems: good and bad parts

Shubheksha Jalan — Sat, 09 May 2020 16:03:33 +0000

Retries are a way to provide resiliency in a distributed system

When working with a distributed system, the only guarantee we have is that things will fail sooner or later. In these circumstances, we want to "design for failure".

Retries are a technique that helps us deal with transient errors, i.e., errors that are temporary and are likely to disappear soon. Retries help us achieve resiliency by allowing the system to send a request repeatedly until it gets a success response. This is useful if you have some component in the path of the request failing the first time around.

There are two ways to retry a failed request:

Manual retries: a failed request prompts the caller which in turn decides whether or not it wants to retry the request
Automatic retries: a failed request is automatically retried by the system without any interference from the caller

For example, imagine a service A needs to talk to service B in order to finish the work it is supposed to do. What happens if service B fails when the request gets to it? We have two options here:

return an error to A and do nothing
return an error to A but automatically retry the request again

If we go down the second route, we can ensure that the system itself can take care of a failed request due to partial failure (C or D failing) without external intervention. This is a super useful feature to have in a distributed system where the probability of something failing at any given time is non-trivial.

Retries can lead to retry storms which can bring down the entire system

Retries, if employed without careful thought can be pretty devastating for a system as they can lead to retry storms. Let's break down what happens during a retry storm with a real-world example.

Consider a queue for a customer service center. The representative can take one phone call every three minutes and the queue of callers keeps flowing smoothly. However, if a few customers are taking longer to be serviced, the rep is much slower than we’d expect them to be when taking calls.

Customers, on the other hand, aren't prepared to wait more than a few minutes and will continue to ring the center from different numbers while being on hold in case the previous call gets through.. This overwhelms the phone line as they can't figure out which call connections should be kept alive and which should be discarded.

A very similar situation can occur within a distributed system as well. Imagine, we’ve multiple services A, C, D and E all trying to talk to service B at the same time. C, D and E are unaware that A is trying to talk to B and vice-versa. If the request from any of A, C, D or E fails, we’ve the following scenarios:

Best case: the retry succeeds in the first or second try
Worst case: the requests can be stuck and will keep getting retried repeatedly if, for example, B is undergoing garbage collection

The worst case scenario can spiral out of hand really quickly if B is being issued lots of requests. All of them will fail and all of them will consequently be retried. It turns into a self-perpetuating cycle where every failed retry in-turn spawns X (X = number of retries your system is configured to use) retries.

We usually don't retry retry requests (meta, I know) as this can lead to exponential growth and bring down a system really quickly. So only the initial failed request is re-tried within the system. Retry requests aren't and shouldn't be issued concurrently, they should be sequential instead in order to avoid increasing unnecessary load on the system.

Let's dig into what happens a little more to clarify it further. B is now being bombarded by different retry requests from multiple clients at the same time while continuing to receive normal traffic from various clients. The load grows linearly over time. This can quickly exhaust B as it will run out of compute and/or memory in order to cope with all the additional load.

Note: We use X=3 in the illustration, but the value of X will vary from system to system, it’s really hard to come up with a one-size-fits-all value for it. However, if you’re retrying requests in a loop, it’s a good idea to have an upper threshold for it which when reached should break out and terminate the request. This will avoid a scenario where we keep trying in an infinite loop.

This situation is known as a retry storm. If we have multiple of these across our system at the same time, then we can end up DDOSing our own system.

It's not easy to detect a retry storm. Doing that will require every node to have a decent picture of what’s happening within the system.

Adding latency can work in our favour

As developers, we're constantly taught that "fast is better", so the idea of adding latency might seem a little weird at first. But it can be really helpful in distributed systems!

However, just adding the same amount of delay between two requests wouldn't help. Let's try to understand why. If, say, a hundred requests fail at the same time and we retry them all with a delay of 10ms, then we're not solving the problem we had on our hands — we just shifted it 10ms into the future.

Exponential Backoff

Another option might be to delay each retry using an exponential delay: for simplicity, let's use 2^n ms delay where n = retry count. Continuing from our previous example, we'll see something like this:

First retry: 2ms
Second retry: 4ms
Third retry: 8ms

It’s always a good idea to have an upper limit for backoff!

So and so forth. Again, this doesn't solve our problem if multiple requests fail at the exact same time within our system (the chances of something like this happening in a real-world system are non-trivial). We'll again issue the retry request at the same time and overwhelm the system. All we've done by this is added delay between successive re-tries without ensuring that they're not synchronised across requests. However, it does give affected larger gaps of time to recover.

Jittering

To break this synchronisation, we can add randomness to the time interval by which we delay retries for a failed request. This is also known as "jittering" the request. In order to simplify understanding, let’s consider the following example:

First retry: 2ms + 0.5 ms
Second retry: 4ms + 0.8 ms
Third retry: 8ms + 0.3 ms

By combining exponential backoff and jittering, we introduce enough randomness such that all requests are not retried at the same time. They can be more evenly distributed across time to avoid overwhelming the already exhausted node. This gives it the chance to complete a few in-flight requests without being bombarded by new requests simultaneously.

The illustrations used in this post are from a doodle I posted a couple of days ago:

Shubheksha ✨

@scribblingon

New comic is all about retry storms!

#devdoodles
#sketchtogether

21:09 PM - 02 May 2020

37 197

Lots of gratitude for Suhail and Ingrid for reviewing drafts of this post. 💜

A few things I wish I knew before I started working as a software engineer

Shubheksha Jalan — Mon, 24 Jun 2019 19:20:11 +0000

18th July will mark two years of my career as a software engineer in the tech industry. Even though I've been coding for a lot longer than that, the majority of lessons I've learnt have come from working as part of a team. In this post, I'll try to distill what I've learnt so far in my journey. As always, let's start with a disclaimer first: this is a recollection of my personal experiences and may or may not work for everyone.

The importance of good leaders:

If there's one thing I've learnt in my rather short career, it's that good tech leads/managers are invaluable. It's an incredibly hard skill set and there are far too many people who suck at it. If you're lucky enough to find good ones, stick with them for a while and if possible, move around with them because they're ridiculously hard to find. They make such a HUGE difference in how fast you progress and directly impact your life so much. Here's what to look for in good leader: kindness, empathy, emotional intelligence. They should be good listeners, be your advocate where possible, help you advocate for yourself and help you do what's best for your career even when it's uncomfortable.

Avoid bad managers/jobs (if you can):

This is controversial point and there are lots of factors at play. If you have the privilege of doing this, I'd strongly suggest not waiting it out at a toxic job working for terrible managers especially if you're early in your career. It actively stunts your growth and hampers your career. In the long run, it does more harm than good in the sense that the effects of a toxic job outlive the job itself. It takes a very long time to get over it and there's a high chance that you'll carry it with you. Bad leaders make you question your worth all the time and you end up feeling absolutely helpless and stuck — it's the classic recipe for burn out.

Building software is so much more than writing code

I realised this pretty quickly but it still bums me out how I had no idea till I started working. Most of the time, the hardest problem is figuring out what code to write and ensuring we're solving the correct problem rather than figuring out how to write it. The latter is much easier than figuring out the former. Estimation and prioritisation is also super hard. Being a software engineer isn't about sitting in a corner and writing code all day. It's an inherently collaborative process that needs a wide variety of skills, especially people skills. It naturally follows from here that being an excellent engineer isn't just about being very strong technically, even though it's definitely super important, that's just one part of the equation. Being an effective communicator, listener and team player are extremely important and valuable skills.

The world is small and the tech industry is smaller

Don't be a dick and treat people badly. This is very simple and logical advice, I know, but I'm still amazed by how many people underestimate how far just being a decent human being will take them. If you do some shit, people will find out sooner or later. The tech industry is much smaller even though it can feel very big sometimes. Word travels faster than you can imagine. Don't underestimate the power of backchannels. If you or your companies treat folks from underrepresented groups like shit, we will find out about it one way or another. The secret underground women in tech cabal meets every couple of days to discuss the shit y'all do.

Repeatedly push yourself out of your comfort zone

All of the good stuff happens when you push yourself to do things you're not very comfortable with or are scared of doing. For a long time, I kept waiting for the "right moment" in order to seize the opportunity and do the thing but it never came. It took me a while to realise that sometimes you've to step way out of your comfort zone even if you don't feel ready (in my case, I know I'll never feel ready) and do it anyway. Wanna start reviewing code? Jump in and review a PR even if you're not very familiar with the code being modified. Ask for help & read it until it makes sense. A little while ago, Sunil told me this and it has stuck with me ever since: write a shit ton of code, especially the bits you’re no good at, and sooner than later you’ll find yourself rubbing shoulders with experts as their peer. I like to compare it to learning to play the piano: if you keep practicing the bits you're already comfortable with, you'll never learn the ones you're not good at.

Invest in building a network early on

I can't stress on this enough, seriously. Nobody told me this and it sorta happened for me accidentally but it's probably the single most important thing I've done for the sake of my career. No matter what people tell you, networking does matter. Treat people with respect and kindness. So many amazing and wonderful opportunities have come my way owing just to the network I built.

As an aside for underrepresented folks: invest in surrounding yourselves with people who are your tireless cheerleaders and will believe in your and your talent even when everything is shit. This industry can be very isolating and lonely on some days and outright shit show on other and it makes ALL THE DIFFERENCE. The best thing I've done for myself in the last two years is surrounding myself with amazing, smart, ambitious and highly technical women I deeply respect, admire and aspire to be. On days the tech industry is a shit show (and it's not an uncommon occurrence at all), that's all that keeps me going. Back in college, all my role models were white dudes and I constantly wondered if someone who looked like me had done amazing things in tech. Now my role models look more like me and are constantly pushing boundaries and paving the way for the rest of us to follow through.

Shout out to Jess Frazelle, Joe Nash and Dizzy Smith for reviewing a draft version of this post. It was originally published on my blog 💖

Re-framing how we think about production incidents

Shubheksha Jalan — Mon, 08 Apr 2019 22:00:00 +0000

I started a new gig 2.5 months ago and it's been quite a fun journey in terms of how much I've had to learn. Starting a new job is not just about learning about technologies and processes, but also learning to adapt mental models you've previously built. It's my second job and everything worked differently from my first one in terms of the engineering culture and processes. We believe in making small and iterative changes and deploying often to production as opposed to doing time bound releases. This has made me think a lot about blameless cultures and how folks early in their career can deal with breaking production (let's just accept that it's inevitable and will happen at some point or another and it's okay)

The goal of this post is to provide some pointers about how teams can help new engineers deal with this and also folks early in their career can re-frame the way think about and deal with production outages.

Re-assurance from senior members of the team goes a long way

Fun fact: the first big change I worked on broke something in production 😂 I didn't know what to do and how to deal with it so, me being me, I started panicking instantly. As soon as it was identified that it was my change that messed something up, my tech lead, Daniel Chatfield sent me a reassuring message. This very simple gesture on his part instantly calmed me down and meant a lot to me and is something that I'll remember for a long time.

Not all bugs are equal

At Monzo, we believe in deploying and shipping small, incremental changes regularly and our platform enables us to do exactly that. As a result, rolling back most changes in exceptionally easy — you just need to run a single command to revert the change.

However, not all bugs are equal. For example, in the particular case I outlined above, that wasn't enough since some invalid data had been written to the database in the time before we detected the issue which needed to be fixed. Being new to the team, I was not super familiar with the service, but one of the more experienced members of the team with more context sat with me to explain how can go about fixing the invalid data calmly and I was able to write a fix confidently. 😊

It's important to design systems and processes that make it difficult to ship large bugs to production. However, bugs will always slip through, and it's invaluable to have tools that allow you to resolve issues quickly and limit the impact. In this case we used our iterator service (that allows us to run jobs across all of our users) to fix the invalid data quickly.

Talk about the stuff you fucked up

This can take the form of in-depth debriefs after an incident depending on how major it is, knowledge share sessions, detailed incident reports, etc. Detailed investigation reports can prove to be super useful here and they can be key in passing context which usually lives inside people's heads to folks new to the team and can fill in gaps in documentation.

At Monzo, we recently started doing weekly lightning talks and one of the first ones was about "Stuff I broke in prod" by Matt Heath, one of our most senior engineers. It's empowering to hear that literally everyone makes mistakes and it's okay. Talking about things going wrong shouldn't be something that fills you with shame or guilt. If that's the case, it's more likely to be an underlying cultural problem and not so much a technical one.

Communication is key

When you know you messed something up, the most important thing is to ask for help. At the end of the day, managing incidents is a team-sport and not a one-person-show. If there's one thing you take away from this blog post, let it be this one. Don't try to hide your mistakes in order to look smart. It's not about you or your ego, it's about fixing the mistake with the least amount of customer impact and you need all hands on deck for that as soon as possible. Communicating proactively is key here and it'll make you a better and stronger team player in the long run.

Treat it as a learning opportunity

Even though this one feels like a no-brainer, I wanna give you some insight into how I like to think about it. When something breaks in production, I usually have mutliple takeaways from it:

I learn to reason about our code better
I learn how to debug the problem (or watch how other people reason about the problem which is also super fascinating for me because everyone's brains work differently)
People actually noticed when something breaks — this is a good thing! That means people use the stuff you build which is cool!
It also gives the broader engineering team insight into how to make the processes more resilient to failures — fixing tooling, updating documentation, automating and detecting failure early, etc., so that the person who comes after you can avoid making the same mistake.

It's never just one person's fault

When something goes wrong, chances are even though you wrote the code, someone else reviewed it. And they didn't catch the bug either. This isn't an invitation to shift the blame on to them for whatever went wrong. It's another way to look at the same thing: humans are flawed and will make mistakes no matter how perfect the tooling or automation. The goal isn't to not make mistakes, it's to learn from them and to avoid making the same mistake twice.

Engineering culture sets the tone for everything

How an individual reacts in such a situation is inherently tied to the team dynamics and the engineering culture of the organisation. If your workplace berates people for making mistakes and singles them out (believe me, this is more common than you think), then chances are very high that you will not feel comfortable talking to your colleagues or peers about your mistakes. This is an overarching point that is tangential to all of the other points I touched. Having a blameless culture where people feel safe to accept that they messed something underpins and is a hallmark of a good engineering team.

I would like to specifically highlight here that it can be really hard to folks in their first or second job to know what a good engineering culture looks like because they don't have have the luxury to fallback on experience. It can leave folks completely disillusioned if their first job is a cultural dumpster fire and yet this isn't uncommon at all. I've written more about toxic jobs and their symptoms in a different post. It is not your fault if you end up in a company with a terrible culture. 💖

To wrap up, I'll try and summarise a conversation I had with a colleague about "humanizing" bugs: instead of treating outages as an invitation to throw an individual under the bus, we should instead try and treat them as opportunity to make our systems better, more robust and more resilient to failure. 😊

Shoutout to Danielle Tomlinson, Daniel Chatfield and Suhail Patel for reviewing an early draft and providing useful feedback and everyone who chimed in with their suggestions in the Twitter thread on this topic! 😊

How To Start Reviewing Code

Shubheksha Jalan — Sat, 16 Mar 2019 22:02:23 +0000

One of my goals at work has been to review code along with writing code. I have been trying to review more and more code and though it's fun, it does come with its own set of challenges. I wanna talk about how I've approached it and what has helped me along the way as I try to get better at it.

It's not rocket science. Seriously, it's not. For a long time, I constantly thought of people who review code as people who have all the context to review the piece of code they were looking at, so of course, they'd find it easy and effortless to do that. This is rarely true, even for smaller companies, with systems that have lots of moving parts. Nobody knows everything about every part of the system. It's good to remember that. Reviewing code is a skill that can be learnt just like writing code. The more you do it, the better you get at it. Reviewing code also helps you write better code because you know what kind of things you need to watch out for. Don't let the fear of not knowing everything about the code you're reviewing hold you back.
Mistakes will be made. Something or the other will always slip through the cracks because humans write code and humans review code and we're all inherently prone to all sorts of errors. That is okay. Just because you can't do it perfectly doesn't mean you shouldn't do it at all. It's also completely okay to ask someone to take a second look if you're not sure about something. Having a second pair of eyes never hurts.

Tangentially related to this point is having a blameless culture where people don't hunt you down if you mess something up in production. The primary focus is to diagnose and fix the problem rather than assigning blame (who wrote/reviewed the faulty code). There is room to make mistakes and what's important is learning from them. I'm lucky to work somewhere that has this culture, so it doesn't feel as daunting. 😊
Code reviews are very ripe for misunderstanding and lack of empathy on either side. At the heart of code reviews is collaboration. It is as important to remind yourself as a reviewer that you're reviewing someone's code and not passing judgments on them as a person and it is equally important to remember that whatever your reviewer tells you is not meant as a personal attack. I feel a lot of this is dictated by the engineering culture of your team and the company overall. Being mindful of this before posting comments can greatly reduce any possibility of friction.

A very simple way to do this:
// Detect dark theme var iframe = document.getElementById('tweet-1103393634739777540-463'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1103393634739777540&theme=dark" }
Another point I want to touch on is the junior-senior dynamic that comes into play when reviewing code. As someone early in their career, it can be very daunting to review code for someone who has a lot more experience and/or context than you. However, this shouldn't discourage you from reviewing their code. Some ways you can do this:
- Shadowing reviews on existing pull requests: this is a great way to learn to spot what kind of things folks watch out for when reviewing code and also how they communicate it and how it is received by the author (how you communicate feedback on a change is equally important, if not more than what you communicate)
- Pairing with another engineer or the author when reviewing something more complex: this let's you ask questions in real time and gain more context about what's going on
Asking for more context is okay. This can be scary at first when you're brand new to a team and/or domain and everyone around you seems to know so much more than you. If something doesn't make sense even after looking the relevant bits and doing your research, it's okay to ask for more clarification.

Let me give you an example of this:
Last week, I was asked to review a small PR for a service I have never touched or used before and my first instinct was to ask someone on my team with more context to do it instead. However, I took it as a learning opportunity and tried to figure out what was happening in the proposed change. After looking at it, even though the overall pull request made sense, some things didn't, so I gathered the courage to state that I'm new to this service and ask the author for more clarification. And guess what - they were more than happy to provide that! Once I had that, I was able to approve their change and at the same time, walked away with several things:
- I'll be more confident about asking for clarification after this positive experience
- I've a better understanding of what that service does and can review related changes more easily
- We identified how we can improve the documentation for the functions we were looking at so that other folks can understand it more easily
It was a win-win in every way! 🎉

Once I started looking at code reviews as just another skill that I can learn and get better at over time, I felt much more confident about doing them. It's less scary now and I look forward to doing more of them. 💪🏼

How To Not Feel Like An Imposter

Shubheksha Jalan — Fri, 14 Dec 2018 22:35:31 +0000

This was originally one of the issues of my newsletter “Life Reliability Engineering”, however I thought it deserved space of it’s own as a blog post.

Credit

Have you ever felt like you don’t know what you’re doing and everyone is going to find out any time now that you’re a complete fraud who has somehow managed to convince everyone you’re smart and know your shit? Welcome to the club, my friend! You suffer from imposter syndrome and you’re one of us! Yay! 🎉

I don’t think it’s a hyperbole to state that the very best of us suffer from imposter syndrome. At this point I’ve come to believe that it’s part of the human condition, especially if you work in any creative field. If someone claims that they don’t suffer from imposter syndrome, they probably have an ego too gigantic to admit it. And you know what? It’s okay to suffer from it.

Credit

I’ve suffered from crippling imposter syndrome over the years, so I wanted to talk about some of the ways I’ve tried to tame it and how it has helped me become a better person and engineer.

Writing your accomplishments down

I can’t even begin to stress how much this has helped me. Having a handy list of tangible professional accomplishments that you can refer to when you’re feeling like a fraud works wonders. Write down your accomplishments — small or big — stuff you’re proud of, stuff you didn’t think you could do but did anyway and things like that. Spoke at a conference? Got a scholarship? Started a new job? Shipped something at work? Write it all down!

Go back to it when you’re feeling like shit and try to remind yourself that if you were a total fraud, you wouldn’t have been able to accomplish all this in the first place. Remember that it being just in your head isn’t enough because your brain won’t re-call it fast enough when it’s in imposter mode, so writing it down is important.

Accepting and reveling in the fact that you’re not the smartest person in the world

(Side note: how, if at all, you can measure this and what does it even mean????) I guess everyone has their own timeline for this, but finally accepting it is, frankly, so relieving. And it has positive side effects! I’m near-constantly striving to work with folks who’re smarter than me and accepting this has enabled me to learn better (and more!) from them! Don’t be that person who is constantly trying to prove they’re the smartest person in the room by putting other people down. Nobody likes that person. Being humble and ready to learn from other folks helps you grow better and faster and also makes you a good co-worker.

Surround yourself with people who recognise your work and worth

This one is a hit or miss in the sense that you don’t really realise the importance of doing this till you actually do it. Spoiler alert: it’s magical. Being surrounded by folks (friends/co-workers/community) who recognise and/or reward your work and reassure you of everything you bring to the table from time-to-time is one of the best things you can do for yourself. It’s truly hard to overstate how big a difference it makes. Invest time in this. It’s very hard to find these folks, but once you do, make sure you stick with them.

Everyone excels at different things and it’s NOT a zero sum game

This point is sort of tangentially related to the second one, but still deserves its own space. There is no super human person out there who is good at everything. That’s simply not possible. Different people have different areas of expertise and that’s a very good thing! It gives all of us something to learn from one another and balances things out. Maybe your strength is someone’s weakness and vice-versa. Someone else being good at something that’s not your area of expertise doesn’t take away anything from you. If anything, it gives you a chance to learn something from them and improve. Recognise and celebrate that.

NOT comparing yourself with people who’ve a lot more experience than you

Comparing yourself to someone who has been around for a long time is kind of like comparing apples to oranges. It’s a recipe for disaster, don’t do it. It’s a bad idea. If you’re someone who is just starting out and compare yourself to someone who has a decade long career, then of course, they’re gonna be doing way better than you! Simply because they’re been around much longer than you have! Of course, they’ve put in a lot more effort than you, built a solid network, spoken at major conferences and what not! (I’m not claiming that everyone does this, but simply picking on of the most common parameters people tend to size themselves up against). This is doomed even before you start doing it. Whenever you start doing this, take a step back, pause and think: is this a valid comparison? Do I need to be comparing myself to anyone at all? Am I happy with where I am? Because that’s what matters in the end, y’know.

Realise it’s a waste of time and energy

I’m a firm believer in healthy introspection, but more often than not, this isn’t healthy. Obsessing over this doesn’t really get you anything when all is said and done. Mostly there are no concrete takeaways and you end up being miserable and down, feeling like a fraud and no matter how much you do, it’ll never be enough. You don’t need or deserve to feel that way. Following this chain of thought is probably not a good use of your time because it doesn’t do anything to actually help you fix anything. Your time will be better utilised in doing or thinking about something more meaningful and actionable. Acknowledging that can at least help you not descend down the “thinking rabbit-hole”.

I can go on writing about this topic endlessly because I’ve battled with a lot over the years and I know it’s not a battle that’s going to end any time (or possibly ever!). So, for now, I’ve been trying to devise ways to deal with it so that I don’t waste time doing this and instead put my energies into being the best software engineer I can. 💪🏼 It has been working out well so far. This has been a long post, but I sincerely hope this will help you deal with random bouts of imposter syndrome as well. ✨

Toxic Jobs and The Tech Industry

Shubheksha Jalan — Thu, 15 Nov 2018 18:58:13 +0000

I did a Twitter thread on this topic a while ago and I’ve some thoughts I’ve wanted to gather for a while on this topic, so I’m combining them into a (hopefully useful) blog post.

// Detect dark theme var iframe = document.getElementById('tweet-1060224805595541504-127'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1060224805595541504&theme=dark" }

You go to work everyday and you want to feel excited but something always feel amiss. Some day you’re full of dread showing up to work, some days you wanna skip it altogether. Rather than being rewarding, it ends up taking a huge mental toll on you everyday. You are scared of what it’s doing to your mental and eventually, even physical health. You try sticking around hoping it’ll get better eventually, but it doesn’t. You wonder if something is wrong with you? Did you manage to screw it up somehow? It ends up being worse if you’ve to deal with this stuff early in your career because you don’t have the luxury of relying on experience to inform your assessment of the situation you’re trapped in.

Credit

This story might sound very familiar to folks belonging to underrepresented minorities in the tech industry.

Let’s dig in and start from the basics.

What is a toxic job?

A toxic job is one that leaves you burnt out and drained every other day. It can cause you trauma affecting your physical or mental health. Constantly feeling tired or dreading going to work are all signs that point to it. Alarm bells should start ringing when it becomes a “normal” occurrence or something you find yourself getting used to instead of something that happens once in a while.

What leads to a job becoming toxic?

It’s very difficult to pin-point a single reason and usually it ends up being a combination of multiple reasons. I can list some of them here:

Unsupportive managers — a manager that doesn’t support you can be a huge source of discontentment at work. Be it supporting your career aspirations in the long-term or day-to-day development, clarity on your immediate and long-term goals, having a supportive manager is key.
Toxic working relationships — a set of coworkers that are negative, completely demotivated and spiteful can also significantly decrease your day-to-day morale at work. Having a healthy amount of mutual accountability, respect and support is paramount for a good working environment. Being deprived of that, being condescending to junior folks or indulging in petty politics in the work place can also lead to jobs being toxic.
Chaotic and disruptive workplace practices — everything around you being in a constant state of disarray is also not very helpful. Worse if nothing is being done to alleviate them and it’s the status quo. Nothing kills your spirit more than feeling that what you do doesn’t matter and there is nothing you can do to make it matter. This can be applied to a large chunk of practices: code reviews, sprint planning, releases, etc.
Uninteresting/non-challenging work — this can depend greatly from person to person, but for some folks if you’re not doing work that’s challenging and you end up making no use of your brain for a continued period of time, it can be the source of a lot of internal commotion. It can easily lead to a depleted sense of self-worth and cause you to wonder “what the hell am I doing with my life” every other night.

Side note: A lot of people think that the main cause that leads to burn out is often too much work when you’ve no time to do anything but work, work, work 24X7. However, the flip-side of it is that having no opportunities for growth, no recognition of the kind of career trajectory you want to pursue and feeling stagnated because you’re not learning anything new and being unable to improve the situation around you can also severely impact someone’s sense of self-worth and in turn mental health paving the way for burn out. This is another blog post altogether.

This is also by research into how humans interact socially.

Credit

Of course there can be multiple other factors, but I’ve tried to list some of the most common ones. The list is way worse and much longer for folks from underrepresented folks who have to deal with a ton of crap just to be able to their jobs.

What can you do if you’re stuck in a toxic job?

Get. Out. Run. No, seriously, especially if it’s affecting your health (yes, even mental health). If you’ve been stuck in a terrible situation like months, it might be time to accept that it’s not you and it’s not going to get better. You can’t fix everything and that’s okay. It’s not your fault, you haven’t failed — it’s your manager/employer who has failed you by not being able to provide a safe and productive working environment. If you can change teams within the company, great, evaluate your options. If not, it might be time to start looking for something new.
Don’t let it become your “new normal” no matter what. Don’t get complacent. This is probably the worst side-effect of a toxic job: it’s so easy to normalize your situation by reasoning around it — “this is how it’s supposed to be” — don’t do that to yourself. It’s not normal. Don’t let it fool yourself and stagnate you for years.
Don’t let it consume you. Surround yourself with people who’ve healthy jobs and are somewhat happy with them. Tell them to remind you that what you’re going through isn’t normal in any way and that you deserve better.
If you’re early in your career and don’t the luxury of falling back on experience, invest time in finding mentors who’ve spent more time here and know better.
Talk to your colleagues if possible. This will be highly dependent on the kind of relationship you’ve with your colleagues, but if possible, you should talk to them about the issues you’re facing at work and ask if they’ve experienced/noticed them as well. This can be a great way to ascertain that there is a much larger problem at hand and it isn’t *you*. This is also a great way to find out what has been done to deal with such problems in the past. If these same problems have existed for years with nothing being done about them, you know it’s a lost cause.
If you’re suffering, please seek help. There is absolutely no shame in asking for help when you’re going through a terrible time. Talk to your friends, family and you’ve access to therapy, please do seek professional help.
Lastly, don’t lose hope. Despite all the crap, there are good people in this industry who’re working hard every day to create a good, inclusive working environment. They’re out there and it gets better, I promise. ❤️

In closing, please take excellent care of your mind and body because:

// Detect dark theme var iframe = document.getElementById('tweet-1062068638755311617-287'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1062068638755311617&theme=dark" }

P.S.- These views are my own and if you don’t agree with them, it’s completely okay.

Robustness in Complex Systems: A Summary

Shubheksha Jalan — Thu, 22 Mar 2018 15:43:29 +0000

Today we look at the paper titled “Robustness in Complex Systems” published in 2001 by Steven D. Gribble. All pull quotes are from the paper.

This paper argues that a common design paradigm for systems is fundamentally flawed, resulting in unstable, unpredictable behavior as the complexity of the system grows.

The “common design paradigm” refers to the practice of predicting the environment the system will operate in and its failure modes. The paper states that a system will deal with conditions that weren’t predicted as it becomes more complex, hence it should be designed to cope with failure gracefully. The paper explores these ideas with the help of “distributed data structures (DDD), a scalable, cluster-based storage server”.

By their very nature, large systems operate through the complex interaction of many components. This interaction leads to a pervasive coupling of the elements of the system; this coupling may be strong (e.g., packets sent between adjacent routers in a network) or subtle (e.g., synchronization of routing advertisements across a wide area network).

A common characteristic that such large systems exhibit is something known as the Butterfly Effect — a small unexpected disturbance in the system resulting from the intricate interaction of various components can result in a widespread change.

A common goal for system design is robustness: the ability of a system to operate correctly in various conditions and fail gracefully in an unexpected situation. The paper argues against the common pattern of trying to predict a certain set of operation conditions for the system and architecting it to work well in only those conditions.

It is also effectively impossible to predict all of the perturbations that a system will experience as a result of changes in environmental conditions, such as hardware failures, load bursts, or the introduction of misbehaving software. Given this, we believe that any system that attempts to gain robustness solely through precognition is prone to fragility.

DDS: A Case Study

The hypothesis stated above is explored using a scalable, cluster-based storage system Distributed Data Structures (DDD) — “a high-capacity, high-throughput virtual hash table that is partitioned and replicated across many individual storage nodes called bricks.”

This system was built using a predictive design philosophy as the one described above.

Based on extensive experience with such systems, we attempted to reason about the behavior of the software components, algorithms, protocols, and hardware elements of the system, as well as the workloads it would receive.

When the system operated within the scope of the assumptions made by the designers, it worked fine. They were able to scale it & improve performance. However, in the case when one or more of the assumptions about the operating conditions were violated, the system behaved in unexpected ways resulting in data loss or inconsistencies.

Next, we talk about several such anomalies.

Garbage Collection Thrashing and Bounded Synchrony

The system designers used timeouts to detect failure of components in the system. If a particular component didn’t respond within the specified time, it was considered dead. They assumed bounded synchrony in the system.

The DDS was implemented in Java, and therefore made use of garbage collection. The garbage collector in our JVM was a mark-and-sweep collector; as a result, as more active objects were resident in the JVM heap, the duration that the garbage collector would run in order to reclaim a fixed amount of memory would increase.

When the system was at saturation, even slight variations in load on the bricks would increase the time taken by the garbage collector in turn dropping the throughput of the brick. This is called GC thrashing. The affected bricks would lag behind their counterparts leading to a further degradation in performance of the system.

Hence, garbage collection violated the assumption of bounded synchrony when it was nearing or beyond the saturation point.

2. Slow Leaks and Co-related Failure

Another assumption made while designing the system was that the failures are independent. DDS used replication to make the system fault-tolerant. The probability of multiple replicas failing simultaneously was very small.

However, this assumption was violated when they encountered a race condition in their code that caused a memory leak without affecting correctness.

Whenever we launched our system, we would tend to launch all bricks at the same time. Given roughly balanced load across the system, all bricks therefore would run out of heap space at nearly the same time, several days after they were launched. We also speculated that our automatic failover mechanisms exacerbated this situation by increasing the load on a replica after a peer had failed, increase the rate at which the replica leaked memory.

Since all the replicas were subjected to a uniform load without taking performance degradation and other issues into consideration.This created a coupling between the replicas and…

…when combined with a slow memory leak, lead to the violation of our assumption of independent failures, which in turn caused our system to experience unavailability and partial data loss

3. Unchecked Dependencies and Fail-stop

Based on the assumption that if a component timed out, it has failed, the designers also assumed “fail-stop” failures, i.e., a component that has failed will not resume functioning after a while. The bricks in the system performed all long-latency work (disk I/O) in an asynchronous way. However, they failed to notice that some parts of their code made use of blocking function calls. This caused the main event-handling thread to be randomly borrowed leading to bricks seizing inexplicably for a couple of minutes and resuming post.

While this error was due to our own failure to verify the behavior of code we were using, it serves to demonstrate that the low-level interaction between independently built components can have profound implications on the overall behavior of the system. A very subtle change in behavior resulted in the violation of our fail-stop assumption across the entire cluster, which eventually lead to the corruption of data in our system.

Towards Robust Systems

..small changes to a complex, coupled system can result in large, unexpected changes in behavior, possibly taking the system outside of its designers’ expected operating regime.

A few solutions which can help us make more robust systems:

Systematic Over-provisioning

When approaching the saturation point, systems tend to become fragile to accommodate unexpected behavior. One way to combat this is to deliberately over-provision the system.

However, this has its own set of issues: it leads to the under-utilization of resources. It also requires predicting the expected operating environment and hence the saturation point of the system. This can’t be done in an accurate manner in most cases.

Use Admission Control

Another technique is to start rejecting load once the system starts approaching the saturation point. However, this requires predicting the saturation point — something that’s not possible always, especially with large systems which have a lot of contributing variables.

Rejecting requests also consumes some resources from the system. Services designed with admission control in mind usually have two operating modes: normal where the requests are processed and an extremely lightweight mode where they’re rejected.

Build Introspection into the system

an introspective system is one in which the ability to monitor the system is designed in from the beginning.

A system which can be monitored and designers and operators can derive meaningful measurements about its operation is much more robust than one a black-box system. It’s easier to adapt such a system to change in its environment, manage and maintain it.

Introduce adaptivity by closing the control loop

An example of a control loop is a human designers and operators adapting the design in response to a change in its operating environment indicated through various measurements. However, the timeline for such a control loop isn’t very predictable. The authors argue that systems should be built with internal control loops.

These systems incorporate the results of introspection, and attempt to adapt control variables dynamically to keep the system operating in a stable or well-performing regime.

All such systems have the property that the component performing the adaptation is able to hypothesize somewhat precisely about the effects of the adaptation; without this ability, the system would be “operating in the dark”, and likely would become unpredictable. A new, interesting approach to hypothesizing about the effects of adaptation is to use statistical machine learning; given this, a system can experiment with changes in order to build up a model of their effects.

Plan for failure

Complex systems must expect failure and plan for it accordingly.

A couple of techniques to do this:

decoupling of components to contain failures locally
minimize damage by using robust abstractions such as transactions
minimize amount of time in failure state (using checkpointing to recover rapidly)

In this paper, the authors argue that designing systems by assuming the constraints and nature of its operation, failures and behavior often leads to fragile and unpredictable systems. We need a radically different approach to build systems that are more robust in the face of failure.

This different design paradigm is one in which systems are given the best possible chance of stable behavior (through techniques such as over-provisioning, admission control, and introspection), as well as the ability to adapt to unexpected situations (by treating introspection as feedback to a closed control loop). Ultimately, systems must be designed to handle failures gracefully, as complexity seems to lead to an inevitable unpredictability.

Nevertheless, Shubheksha Jalan Coded

Shubheksha Jalan — Wed, 07 Mar 2018 18:23:51 +0000

I began to code because...

I've been fascinated by computers and how they work from a very early age. I wanted to know how and why they work. I remember searching for it on my father's laptop when I was in 6th grade. A bunch of random stuff came up, including a PDF with a ton of complicated stuff. I didn't understand any of it then, but that's when I decided to dig deeper and find out someday. My first tryst with computers was in the form of algorithms in 8th grade. I just fell in love with how logical it was. I found myself hooked on it super quickly. From there, I went on to take CS as my elective in 9th grade where we were taught Java. The first program I ever wrote, a "Hello World" obviously -- when it worked after a bunch of silly syntax errors, my joy knew no bounds. I still remember the feeling very distinctly and till date, it is one of my most cherished memories.

I recently overcame...

This is a tough one. I overcome something almost every other day. Mainly it's anxiety, panic, fear, something along those lines. In the past few months, I overcame my fear of reading research papers. Despite having a CS degree, I always thought it was not for me since I don't have a research background (not sure what that means exactly). I dove right into reading them because I wanted to learn more about distributed systems. Not only that, I also started publishing summaries to help make it more approachable for other people looking to start.

I want to brag about...

I really enjoy drawing doodles to illustrate technical concepts better. I'm looking forward to doing more of that this year too.
As mentioned before, I also publish summaries of research papers every now and then.

Apart from this, I did a couple of great things last year:

Travelled to three countries for conferences through scholarships
Did two internships, one with Mozilla & Outreachy
Gave my first conference talk - Open Web, Open Source, Open Communities
Started a new job at Microsoft!

My advice for women and non-binary folks who code is....

One thing I’d like to tell everyone out there, something I strongly believe in — don’t conform. No matter what happens, do your own thing. If it goes right, the benefits are yours to reap. If it goes wrong, you’re to be blamed and you can’t pin it on anyone else. Don’t listen to what the society has to say about how women should & shouldn’t behave. It’s your life, live it ONLY on YOUR own terms, not someone else’s. Don’t try to fit into a someone else’s description of you & your life. If your career & financial independence is of utmost importance to you, give it everything you got & don’t let anyone make you feel bad about it just because you’re a woman.
Every time I feel discouraged by a sexist comment someone randomly tries to pass as a "joke" or someone tries to discourage me because of my gender, I just think of the reason I got into tech and why I am here -- the idea of building things that can be useful to people with just words, i.e., lines of code is utterly exciting to me. I want to help simplify the lives of folks around me through code. I want to be a badass software engineer and I'm not ready to back down anytime soon.

An Introduction To Docker Tags

Shubheksha Jalan — Mon, 12 Feb 2018 17:55:45 +0000

Credit

If you’ve worked with Docker even for a little bit, I bet you’ve come across tags. They often look like “my_image_name:1” where the part after the colon if known as that tag. It’s not always specified when tagging images, we’ll get to the bottom of that later.

Ever since I started using Docker, I’ve been very confused about tags. The docs don’t help in understanding them. Neither are there thorough explanations on the topic. Which is why I decided to write this post.

What are Docker tags?

So…what exactly are tags? In simple words, tags are just aliases to the ID of your image which often looks like this:f1477ec11d12. It’s just a way of referring to your image. A good analogy is how git tags refer to a particular commit in your history.

The two most common cases where tags come into play are:

When building an image, we use the following command:

docker build -t username/image_name:tag_name .

Let’s try to unpack what this command does for a bit. We tell the Docker daemon to fetch the Docker file present in the current directory (that what the .at the end does). Next, we tell the Docker daemon to build the image and give it the specified tag.ow, if you run docker images, you should see an image whose repository is username/image_name and tag is tag_name.

username/image_name is not a mandatory format for specifying the name of the image. It’s just a useful convention to avoid tagging your images again when you need to push it to a registry. Your image can be named anything you want. For the public Docker registry, you’re restricted to a two-level hierarchy while naming images, i.e., your image cannot have the name a/b/c:1. This restriction usually doesn’t exist in private registries. As stated before, it’s not mandatory to specify a tag_name. We’ll see what happens in that case soon.

Explicitly tagging an image through the tag command:

docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]

This command just creates an alias (a reference) by the name of the TARGET_IMAGE that refers to the SOURCE_IMAGE. That’s all it does. It’s like assigning an existing image another name to refer to it. Notice how the tag is specified as optional here as well, by the [:TAG] .

What happens when you don’t specify a tag?

Alright, now let’s uncover what happens when you don’t specify a tag while tagging an image. This is where the latest tag comes into the picture. Whenever an image is tagged without an explicit tag, it’s given the latest tag by default. It’s an unfortunate naming choice that causes a lot of confusion. But I like to think of it as the default tag that’s given to images when you don’t specify one.

A lot of confusion around latest is caused due to the expectation that it’s the latest version of the image, especially in Dockerfiles. Let’s consider the various scenarios with an example:

Scenario 1:

Suppose we’ve the following statement in our Dockerfile:

FROM debian

Since we don’t specify any tag, Docker will add the latest tag and try to pull the imagfe debian:latest .

Scenario 2:

FROM debain:9.3

Since the tag is explicitly mentioned here, Docker will pull the Debian image tagged 9.3

Another thing to keep in mind is that there is no rule which states that an image needs to have just one tag. An image can have multiple tags and they’re usually used to specify major and minor versions. For example, consider this:

Docker Hub page for Debian

At the time of writing of this post, the latest tag for the Debian image points to the 9.3 release AND the 9 release. This will most likely change in the future whenever the major or minor version is bumped for the image.

Please note that tags being used for semantic versioning is a convention that’s followed, but tags weren’t designed just for that purpose.

In conclusion, latest in not a special tag

The main takeaway from what we’ve covered so far is that: latest is just like any other tag. The onus is on the developer to tag the images properly such that latest always points to the latest stable release of the image.

Hence, if don’t explicitly specify a tag in our Dockerfiles when pulling images, the next time we pull our image, we might end up with a completely different version of the base image that what we had used before. There are no guarantees about whether it’ll be a major bump or minor bump. Even an old release can be tagged as latest.

P.S. — If you find any misconceptions/errors in the post, please feel free to tweet to me @ScribbingOn.

Thanks to Jérôme Petazzoni for helping me make sense of some of this.

Program Design in the Unix Environment: A Summary

Shubheksha Jalan — Thu, 08 Feb 2018 19:24:18 +0000

Credit

Today, let’s take a look at "Program Design in the Unix Environment” published in 1983 by Pike and Kernighan.

The paper opens by listing why Unix has been successful and is a commentary on the Unix philosophy and its benefits. It does so by taking examples and discussing trade-offs where programs diverged from the Unix philosophy.

The reasons for Unix’s success:

Portability: the kernel & applications are written in C, hence they can be moved from system to system without being re-written in the assembly language particular to that system.
The same OS runs on different hardware, so the users are already familiar and don’t have to relearn when new hardware is released.
The vendors can ship same software with each machine despite changes in hardware.
The system was not too big and easy to modify since everything was written in C.
It provided a new philosophy based on the use of general purpose tools which did one thing well and could be combined to do a particular task instead of creating giant monolithic tools that serve only one purpose.

The paper argues that the use and design of tools are closely related — how they fit together is the main subject of this essay.

The paper then dives into cat— the Unix command line utility for concatenating and printing files — it copies its input to its output. The input is usually a sequence of one more of files or the standard input. The output is a file or the standard output.

The main purpose of cat was to act as a utility to concatenate files. It can be combined with the pipe (|) operator to further enhance to extend its utility through output redirection.

Other systems, on the other hand, try to dump a bunch of related functionality into a single command which is against the Unix philosophy. It also creates a lock-in of functionality that might to useful to other programs.

Advantages of the Unix approach:

The shell and the programs it can invoke provide a uniform access to system facilities. Eg: the filename arguments are expanded by the shell in a similar fashion for each command. Because of pipes, we don’t need every command to deal with pre- and post- processing of input.
Growth is easy when functions are well separated.

Example: the ` (backquote) operator was added to convert the output of one program into an input of another without requiring changes in any other program as it is interpreted by the shell. All programs the shell invokes acquire this feature automatically. If each program that required this feature interpreted it, it’d be very hard to enforce uniformity and carry out further experimentation as each new idea would affect all the programs that would want to use it.

However, in the future versions of cat, many new options were introduced, like for printing line numbers and non-printable characters.

The authors argue that instead of adding those options to cat itself, either existing programs should’ve been used or new programs should’ve been created. For example, line number functionality could’ve been provided by using pr. However, there was no program which allowed printing of non-printable characters hence warranting the creation of a new one.

Such a modification confuses what cat’s job is concatenating files with what it happens to do in a common special case showing a file on the terminal. A UNIX program should do one thing well, and leave unrelated tasks to other programs. cat’s job is to collect the data in files. Programs that collect data shouldn’t change the data; cat therefore shouldn’t transform its input.

Whenever we split something into multiple programs, we sacrifice some efficiency. But since cat is usually used without any options, it makes sense to have the most common cases be the most efficient.

Separate programs are not always better than wider options; which is better depends on the problem. Whenever one needs a way to perform a new function, one faces the choice of whether to add a new option or write a new program (assuming that none of the programmable tools will do the job conveniently). The guiding principle for making the choice should be that each program does one thing. Options are appropriately added to a program that already has the right functionality. If there is no such program, then a new program is called for. In that case, the usual criteria for program design should be used: the program should be as general as possible, its default behavior should match the most common usage, and it should cooperate with other programs.

Let’s consider another issue: dealing with fast terminal lines. How to deal with output from cat scrolling off the top of the screen?

There are two approaches:

Tell each command about terminal properties so it does the right thing
Write a command that handles only terminals without modifying other programs

Let’s consider examples of both approaches: lsc and ls which prints out the list of files in a directory.

lsc varies its output depending on the input — it displays the list in a columnar fashion across the screen so that the o/p fits if it’s outputting to the terminal whereas ls displays everything in a single column.

By retaining single column output to files or pipes, lsc ensures compatibility with programs like grep or wc that expect things to be printed one per line. This ad-hoc adjustment of the output format depending on the destination is not only distasteful, it is unique no standard UNIX command has this property.

The authors argue that the columnation facility is useful in general & shouldn’t be locked away in just lsc and be inaccessible to other programs. They advocate for a different program whose primary job is columnation.

Similar reasoning suggests a solution for the general problem of data flowing off screens (columnated or not): a separate program to take any input and print it a screen at a time. Such programs are by now widely available, under names like pg and more. This solution affects no other programs but can be used with all of them. As usual, once the basic feature is right, the program can be enhanced with options.

Based on the previous example, the authors also talk about different cases where some functionality is locked away in a specific program, like input history in the terminal, which would be better off as a central service. All interactive programs could benefit from it.

They conclude with how augmenting existing commands with features/options are not desirable in Unix, it goes against its basic philosophy which is to make a program do one thing well and several such programs can be composed to accomplish a more complex task.

The key to problem-solving on the UNIX system is to identify the right primitive operations and to put them at the right place. UNIX programs tend to solve general problems rather than special cases. In a very loose sense, the programs are orthogonal, spanning the space of jobs to be done (although with a fair amount of overlap for reasons of history, convenience or efficiency). Functions are placed where they will do the most good: there shouldn’t be a pager in every program that produces output any more than there should be filename pattern matching in every program that uses filenames.

One thing that UNIX does not need is more features. It is successful in part because it has a small number of good ideas that work well together. Merely adding features does not make it easier for users to do things it just makes the manual thicker. The right solution in the right place is always more effective than haphazard hacking.