DEV Community: Modasser Billah

Going Remote for Life

Modasser Billah — Sun, 06 Sep 2020 22:02:12 +0000

Remote work is trending. But the remote revolution started long before the pandemic and will stay powerful even after the hype subsides. As with any change, how well you adapt and improve will determine whether you get the better of remote work or run back to offices once the external factors go away.

I decided to go remote 3 years ago and I'm glad I did. My last workplace was BriteCore, a remote-first company based in the US. And tomorrow, I'll be joining Doist, a pioneer in the world of remote work. I have gone through a lot of interview processes in the recent years.

This article is an attempt to answer the why, what and how of remote work. The intended primary audience are developers from developing countries like me (I'm from Bangladesh) who do not want to relocate right now but wants to work on exciting products and teams all over the world. The general ideas apply to all professions that can be carried out remotely and to professionals anywhere in the world.

Why Remote

Better Opportunities

Going remote means your employment is not limited to your physical location anymore. That opens up a world of opportunities. You can work for your dream company or with the colleagues and experts that you cherish. And all of it while staying in the place you like with the people you love. If you are the nomad type, this gives you great flexibility.

Better Financial Gain

💵 Remote work opens up much better opportunities with much better compensations if you're, like me, living in a developing country. The compensations are generally much better than the local industry.

📈 The high exchange rates between the powerful currencies and the local ones allow you to leverage the asymmetry and save much more. In fact, you can end up with a higher net savings per month than someone earning more than you in expensive places like San Francisco or London for example.

📉 Tax rates are also generally lower and most developing countries have generous rebates if you're bringing in foreign remittance.

✅ Even if you're living in a developed country, remote work saves you a good amount from commute, lunch and everything in between that comes with showing up in an office.

Better Self-actualization

Remote work allows you to do deep work and puts you in control of your routine and interruptions. In a world of constant connectivity, the option to go deep as needed can give you an unfair advantage in knowledge based work that reward craftsmanship and superior skills. Read the interesting book titled Deep Work by Cal Newport to learn more about this.

The Remote Skillset

Remote work comes with its share of challenges. You need to adapt and overcome these challenges to thrive in a remote work environment for the long term. And the process starts with adopting the remote skillset.

I use a handy mnemonic to summarise the skillset essential for remote work. I call it the IBM CI. IBM, as you probably know, is the name of a famous company and CI or Continuous Integration is a term developers are quite familiar with.

The mnemonic comes from Independence, Balance, Mastery, Communication and Impact. I borrowed it from Doist. So, all credit goes to the awesome team there. I just came up with the mnemonic to remember it better. You can find it in any of their job descriptions. Their blog is chalk-full of great articles on remote work. Subscribe to the blog to get the best out of it.

I'll only touch on two aspects that encompass everything else.

Manager of One: Going remote means you're unlikely to have colleagues by your side. If you're working from home, it can be quite a challenge to put a boundary between work and non-work time. You're in charge of yourself and your schedule. So, you need to be able to do it well if you want to succeed in remote work in the long term.

I borrowed the term Manager of One from Basecamp, a pioneer and advocate of remote work. The ability to manage yourself is pivotal to a successful remote career. Read more about it on Basecamp's blog post on the topic.

Communication: Bulk of the communication in remote work happens in the written medium. You need to harness your ability to express your thoughts and ideas clearly in the written form to thrive in a remote setting. This is also going to be the primary mode of making connections with colleagues across the company. You don't need to become a literary genius though. Clear and concise technical writing goes a long way. This tweet sums up a lot of great points.

The tone gets lost in transmission in written medium. It's always good to pay extra attention to not come off harsh. Warm your messages with emojis :)

Most remote settings will have a diverse workforce. It's important that you stay sensitive to cultural diversities and refrain from saying anything that may be acceptable in your culture but not in others. Avoid using swear or curse words in all settings even if you deem it to be informal. I have had a manager in the past who would use the f-word in meetings. Please don't!

As for technical skills, I advice to work on-site when starting out if possible. Most remote jobs expect you to do your job with minimal supervision. It's easier to find jobs and perform better when you have some experience under your belt.

The Remote Toolbox

Remote work means managing yourself well. Most of your communication will happen online and it is expected that you'll be using a lot of collaboration tools. Here are some of my suggestions to organise yourself better and produce quality output.

📝 Use a task management tool. Put all your tasks as they pop up in your list and plan them accordingly. I use Todoist for this but any tool that you're comfortable with works, even pen and paper.

📓 Use an app for writing and taking notes. I prefer iA writer for long documents and Evernote for quick notes synced across devices. But again, anything that you're comfortable with works fine.

🛠️ Familiarise yourself with communication tools like Slack, Teams, etc if not already. Most companies use one for communication. While there's not much of a learning curve, you should at least know that they exist. Same for project management tools like Jira, Trello and so on.

Shades of Remote

This section is more focused on developer jobs. Remote work can mean a lot of things for a developer. It can be consulting, freelancing, working for steady clients or full time contracts with companies. I'll categorise them in 3 buckets and list some resources where you can find opportunities for that category.

Freelancing Platforms

You can freelance on projects remotely on freelancing platforms. Examples include Upwork, Freelancer, etc where you can join, create a profile and start bidding on projects.

While this is a good option to work on projects you're interested in, I tend to prefer not working on them as a career choice. All clients are not the same, so it needs a lot of calibration every time you start a new project.

Clients generally choose these platforms for cost optimisation. So, the engineering standards are generally a bit lower and the deadline pressures are more intense.

This also means variable workload and variable income, which makes planning your time and life harder. But I have worked on projects intermittently on Upwork before and it can be a good path to pursue if it suits you.

Here's a list of popular freelancing platforms:

Agency Platforms

Some platforms employ a vetting process and only allow successful candidates on their platforms. For the lack of a better word, I call them agency platforms. These platforms are on the rise and I see new ones every now and then.

Once you're on the platform, you may have to bid for projects again or you may be paired with companies depending on the type of the platform. But you'll be working as a member of the platform and the platform will take a cut off your earnings.

One major downside of both these types of platforms is that you don't get any paid time offs (PTO). If you don't work, you don't get paid. You don't get any other perks or benefits either. But once you get yourself on these platforms, it may be easier to land clients or contracts (and better ones) than on platforms mentioned in the first category. Remember to check reviews before joining. Do not join if you don't like their practices.

Here's a list of some platforms in this category:

This list is not exhaustive and there's a lot of other platforms like these. I haven't worked on any of these, so do your review before you decide to join.

Remote Companies

Remote-first or full-remote companies fall in this category. The number of such companies is increasing rapidly, thanks to the pandemic. You can apply to these companies directly from their job posts and work as independent contractors after you successfully complete their recruitment process.

I prefer this category of remote work because you get to work for the company directly. Generally, the teams and cultures are better in companies who hire directly.

Similarly, the compensations and benefits are better as well. Most of them will provide PTOs and have good practices around remote work.

Some popular companies that hire remotely are:

There are many job forums and aggregate websites that you can subscribe to for job openings in remote companies. You can create email subscriptions according to your choices. Here's a list of some popular remote job forums:

These platforms are great for job search. Make the best use of them by subscribing to filtered job searches so that you get alerts for jobs that suit you.

Getting Hired

Cast a Wide Net

You can find tons of job postings on the internet. But the sad reality is most of them won't reply back to you. And you'll only proceed to final stages on a few of those that do reply. So, you need to apply to a lot of jobs to interview at a few of them and then eventually get offers from a handful of them. Don't get discouraged by rejections or no replies, instead keep applying to opportunities that you like.

Don't refrain from applying if you feel you don't fulfil all the requirements for the role. As they say, if Internet Explorer can ask you to be your default browser, you can apply to jobs that you think you don't qualify for. If you never try, you'll never know.

Focus on a Niche

This applies two-fold. Firstly, choose a set of technologies or industries that you like and preferably have experience in. And apply for jobs only in that category. For example, I tend to focus on roles that focus more on the backend and requires Python and AWS skills.

There's an ocean of platforms and frameworks out there. Getting the basics in place is important. But you can get better results if you focus on certain skills, languages and technologies while starting a new job search.

Secondly, companies generally have two major modes of recruitment. One type of companies will send you timed coding challenges and require live coding interviews. You need a more Cracking the Coding Interview style preparation for this type of recruitment processes.

The other type of companies will have a more thoughtful recruitment process suited to remote work. They'll test your communication skills with descriptive writing tasks and behavioural interviews with some technical questions to gauge your technical depth. Then they'll pair it up with take home assignments that match closely with the work you'll be doing for the company if they hire you. I prefer this process and tend to focus more on companies that follow this process.

Pick your battles wisely and prepare for them accordingly.

Getting the Foot in the Door

Cover letter and CVs get your foot in the door. This is generally the first step of the recruitment process. Make sure you pass this initial screening by tailoring your CV according to the job and a thoughtful cover letter.

Most applicants ignore the cover letter altogether or send a generic one. Stand out by crafting a great cover letter that shows your interest in the company and the value you offer for them and the role. There are tons of resources on this on the internet for free. Make good use of it. Here's a great article to get started on the overall process.

If you're applying to a company you like, make sure you let them know that you admire them and the reasons for it. I admire Doist and have been using their product for some time. Their blog has also been a source of inspiration for me. So, I made sure to make a note of this in my cover letter.

Do Your Research

Once you hear back after applying, do some research on the company. Read the glassdoor reviews on the company and look for interview reviews as well. Go through the company website, follow key persons of the company and employees in the department you're applying for on social medias.

If the company has a blog or some of their employees have been on podcasts and webinars, spend some time on those. This will give you a lot of perspective on the company and the culture and help you come up with some good questions to ask during the interviews.

Read the job posting carefully and list out the important technologies and concepts they emphasise on. Focus on these during your technical preparation. Glassdoor interview reviews and the job posting can offer you important insights on the interview process. Take advantage of them to gain an edge.

One unconventional strategy is to find ex-employees of the company online and ask them for advice. Most people add work histories on their social media profiles and you may get lucky finding some of them.

Most people are generally friendly and helpful and they can give you valuable advice on how to prepare. You can also talk to them about the culture and environment in the company to understand whether it suits you. Remember, you're interviewing the company as much as they are interviewing you.

I actually did this when I found an ex-doist employee on LinkedIn, Ahmet. He was very helpful and his inspiring words and valuable advice helped me a lot to prepare for the interviews at Doist.

Avoid Unforced Errors

Prepare for the interviews well.

👉 Choose a quiet place.

✅ Have a backup internet plan if the power goes off.

📷 Always keep the camera on even if they make it optional. Face to face interaction is very important and don't waste the opportunity.

✨ Make sure the background is nice and you're dressed for the occasion. You don't need to don a suit though, just don't look like a shabby hacker.

🗣️ Communicate well all along the interview process. Reply to emails in a friendly manner.

😄 Smile a lot during the interviews, it does wonders for you and creates a positive impression on the interviewer.

Nail the Take Home Assignments

It's easy to think doing what is asked is good enough. But if you really want the job you'll go the extra mile. Yes, it's important to do what you're asked to do, but it is equally important to do it in a way that showcases your skills. Here are some simple yet overlooked advices for test projects:

Use a syntax formatter.
Follow a style guide (e.g, PEP8 for Python).
Name your variables well.
Write some tests. These don't need to be exhaustive but cover the most crucial parts of your code at least. This shows that you care about it.
Write docstrings if possible, even if one-liners.
Use comments thoughtfully to explain the why but not the how.
Write concise and meaningful commit messages. Squash commits to present a coherent and polished workflow.
Ask clarifying questions. If you make assumptions, try to write them up in a markdown document along with your approach to the given problem.
Don't go overboard with your solution. Don't complicate it or add features that are not asked for. How well you understand the problem specification and implement accordingly matters a lot.
It's a test project for a reason, so don't try to cover all aspects of a system, instead focus on solving the major problems. Do it in an organized way and then write up a document on the improvements that can be made on further iterations on the problem.

Grow from Rejections

Rejections will come across aplenty. Recruitment is something no one has figured out precisely yet and you can be rejected even if you're qualified. So, don't lose heart for getting rejected.

People rejected by big tech have done wonderful things going forward. If the recruitment process uncovered a shortcoming on your side, maybe a gap in your knowledge, make sure you don't make the same mistake again.

Interviews have no fixed boundaries and you can be asked literally anything. So, don't be too hard on yourself if you don't get all the questions right. Look them up and learn. If you're learning from every experience that means you're growing and increasing your probability for success.

Staying Employable

So you did it and got your dream job. Don't slack off now and be too indulged in your success. Getting the job is only the beginning of the challenge. Onboarding in a new team and codebase is more challenging than the interview process. You have a point to prove too.

So, stay focused with a growth mindset and make the most of the opportunity to learn and grow in your role. Have personal learning goals and invest in continuous learning. Absorb knowledge from colleagues and help people around you.

Did You Know?
💡 ACM offers special membership rates for members in developing countries. If you become an ACM member, you will get access to O'Reilly Learning. This can be a great deal for you. Check out rates for your country here.

You can be putting in a stellar performance and still lose your job. The pandemic has demonstrated this very clearly. So love your job but be prepared to get back on the market. Change is the only constant in life. If you have the right mindset, you can turn adversities into opportunities.

Develop a Digital Footprint

If you like to write, start a blog. If you don't enjoy writing, start a blog anyway 😁

Here's how you can get into the habit:

💎 Put up short posts on the things you learn or issues you solve in your projects.

📝 Take notes from books you read and post them.

The benefits of writing will compound with time and your future self will thank you for it. It also gives you a positive exposure to potential employers and clients.

Build Your Network

Networks are always the best source of leads and referrals. Make sure you grow meaningful connections along the way. Here's how you can start growing your network:

👨‍🚀 Follow experts and veterans in your industry on social medias. I learnt about the opening in BriteCore from Daniel Feldroy's tweet. It worked out wonderfully well, I worked under him for a while at BriteCore and recently we started bd.feldroy.com together! [Update: we had to shut down due to the pandemic, but you get the point.]

🔖 Subscribe to blogs and newsletters you like. Interact with them and grow your network.

🔧 Contribute to open source projects if possible. This is also a great way to build your network.

👥 Don't forget to network with your colleagues in the company. Talk to people in different departments, connect with them on social media. Connections you create along the way are priceless.

Keep It Simple

It's been a long read. If you've read this far, that means you're quite convinced to go remote. All this information can be overwhelming and there's a whole lot more out there on the internet. So, I'll end with a note to keep it simple and focus on a few key things.

🔮 Mindset matters. If you think you can or can not do it, you're probably right.

🔎 Focus gives you an edge. Stay focused on your goal. Don't run after shiny new things and change course too often.

✊ Be Resilient, better yet be antifragile.

The resilient resists shocks and stays the same; the antifragile gets better.
-Nassim Nicholas Taleb

📚 Read voraciously, in and out of your area of expertise. There's no better way to leverage what other smart people have figured out already.

Finally, don't take yourself or the job too seriously. Enjoy the nice things in life and have a balance between your ambitions and personal life.

You are not your job, you're the person you are.
-Toni Morrison

If you have questions or want to talk to me, feel free to reach out to me on twitter @billah_tishad

Now go get that job and say goodbye to office spaces!

Book Review-The Temple of Django Database Performance

Modasser Billah — Wed, 05 Aug 2020 18:51:56 +0000

I recently bought an interesting tech book themed on fantasy adventure. And it was money well spent. The Temple of Django Database Performance was a great read.

The only way to master Django is to master database performance. The difference between junior, mid-level and senior Django engineers is usually a function of this mastery.

The author starts his book with the bold claim above. But I'd say the content applies beyond Django and helped me get more low-level with databases than before. I encourage django devs to read this book. If you need a glimpse into it, here's my notes from the book:

Profiling: The Yardstick

Before you start your ritual of performance tuning, you must be initiated in profiling. In human terms, profiling is all about measuring your results. Without profiling you have no way to measure if your tuning pleased the database deity.
APMs or Application Performance Monitoring tools are your friend here. Think New Relic, Datadog or more recently Sentry. The book touches on details in New Relic, but I'll skip that part as that is too application specific. You can get up to speed with any of these quickly.
Slow Query Logs can help as well but not at the level of APMs.
For locally producing issues and gauging performance, use Django Debug Toolbar and/or Silk
Query Plan: The query plans details how a database intends to execute a query. Get comfortable reading these. The book is chalk-full of examples of query plans.

Indexes: The Magic Wand

Without indexes, databases have to do sequential searches every time. O(n) every time, oh no! 😟
B- Tree to the rescue. Indexes are generally B-trees. B-tree maintains a tree whose leaf nodes are pointers to the disk locations of table records.

Fun fact
🛩️ Engineers at Boeing invented the B-tree data structure. But no one knows what the B in B-tree stands for.

Databases store data in units called pages. Page size is also used for index storage. In theory, the speed of the index is proportional to page size. But it has a lot of external factors like storage medium and application. So, when in doubt or in the mood to tinker, profile!

Postgres Tip
📌 You can see indexes defined on a table by executing \d <table name> command in the psql shell.

❓ Django ORM question: How do you add an index to your Django model?

💡 Specify indexes field inside the Meta class:

Class Event(models.Model):
    ... # omitted for brevity
    name = models.CharField(max_length=255)
    ...
    ...
    class Meta:
        indexes = [
                    models.Index(fields=['name'],
                    name='my_first_index')
                ]

Note
🔵 The name attribute of the index is optional but helpful for later use.

Postgres Tip
🔓 Postgres locks the table for write operations by default during the indexing process. This can take a long time on large tables and cause headaches for your live system. Way around? Use, CREATE INDEX CONCURRENTLY

☝️ This is called an Online DDL operation.

DDL: Data Definition Language => modifies the schema of the database.
DML: Data Manipulation Language => acts on data
In relational databases, both DDL and DML are just two tribes of SQL commands.
MySQL InnoDB engine doesn't lock by default.

Postgres Tip
💎 Force it to update statistics by VACUUM ANALYZE, then check performance with EXPLAIN ANALYZE

🚨 In general, indexes slow down write operations.

For write-heavy applications, you need LSM Trees or SSTables instead of B-trees. Checkout Cassandra and RocksDB for these. If you're using Django+MySQL, check out MyRocks from Facebook.

Most columns do well with B-tree indexes. However, if you are storing exotic data, you may benefit from alternative index types

GIN indexes for JSON data, Postgres
GiST indexes for spatial data, Postgres
BRIN indexes for very large tables, Postgres
Hash indexes for in-memory tables, MySQL
R-tree indexes for spatial data, MySQL
GIN = Generalized Inverted Index
- Works well with contains logic for JSON data and full-text search
- Only available in Postgres
Indexes are not a silver bullet (well, almost)
The query planner may ignore an index and not use it! 😮

❓ Why?

💡 There can be many reasons. For appetisers, check your query plans after introducing an index to confirm whether the query plan chose to use the index.

Tuning parameters/configs of the database can help, too. This article has some nice pointers about this in Postgres.
If everything fails and indexes don't heal you, it may be a sign that you need to move some work out of the database and into your application, denormalise or move it into an asynchronous task.
Covering Indexes: An index is said to cover a query if the database can return all the data needed for the query directly from the index, without having to search through the table data.

❓ How do you create a Covering Index?

💡 Multi-column indexes

CREATE INDEX CONCURRENTLY le_index_name
ON le_table_name(le_attribute_1, le_attribute_2);

Note
❗ Order of the attributes matter! An index on (id, name) is not the same as the index on (name, id)

💡 Using the INCLUDE parameter

CREATE INDEX CONCURRENTLY le_index_name
ON le_table_name(le_attribute)
INCLUDE (le_another_attribute);

Partial Indexes: An index with a WHERE clause is a partial index. It limits which rows in the table get added to the index.

CREATE INDEX le_index_name
ON le_table_name(le_attribute)
INCLUDE (le_another_attribute)
WHERE le_attribute = 'le_frequently_used_value';

Clustering

If the table changes infrequently and most of your queries filter on the same set of columns, you have another option: Clustering.

Clustering restructures the table data on disk in accordance to the structure in the index.
Clustering is a one-time operation. Postgres does not maintain the structure of the table, so the structuring gets violated with new writes.
It also requires an ACCESS EXCLUSIVE lock on the table, which blocks reads and writes.
So, before you say I do to Clustering, read the quote above again and decide whether you should.

MySQL Tip
💎 Tables are automatically clustered by primary keys in MySQL InnoDB engine. And no, you can not cluster by a secondary index.

Querying: The Weapons and the AMMOs

The n+1 problem
- 1 query fetches n records
- Then you do 1 query for each of those records to fetch some related data.
- 1 (the original ~~sin~~ query) + (One additional query for each record) 1 * n = n + 1
Healing potion?

💡 Django uses lazy fetching, that's why it doesn't fetch related models in the original query.

💡 Force it to be not lazy, use select_related.

select_related makes a JOIN to get data for each field name you pass in.
```
events = Event.objects.select_related('user').all()
```

Note
❗ Joins can negatively impact performance. So, like any other super power, use select_related with care and only when you need to.

select_related only works with single value relationships like ForeignKey.

🤔 What about many-to-many and many-to-one relationship (i.e, reversing a ForeignKey relationship)?

💡 prefetch_related to the rescue!

User.objects.all().prefetch_related('events')

Relevant Resource
📚 This article is a great place to start on select_related and prefetch_related.

More ammunitions:
- Use only() to specify which columns/attributes of a table/model to fetch
```
User.objects.only('name')
```
- defer() is the opposite of only(). Defer fetching all columns that you don't need immediately.
- values() fetches data without instantiating any models. You get the data as a dict.
- Consider using values() when you're willing to trade Django models for more speed and lower memory usage.
- For large set of data use iterator() to read data in chunks.
- Use update() when
  - Updating all objects in a query set to the same set of values
  - The update does not need any signals or custom save() methods.
- use bulk_update() when
  - The query set is large
  - Objects in the query set are updated to different values
  - The update does not need any signals or custom save() methods
- F() expressions allow you to refer to a value of a column in the database without querying for that data.
- func(), annotate, aggregate are some more advanced weaponry. Skipping details, just remember they exist and pick up as needed.

Relevant Resource
📚 Haki Benita's blog has some great articles on databases, Django ORM and Python.

Pagination

Pagination if often overlooked when building a new system. But it bites you eventually when the data grows in size.
- Offset pagination
- Keyset pagination a.k.a infinite scroll
Offset Pagination
- Django comes with a Paginator class that does offset pagination off-the-shelf.
- Flaws
  
  ❌ Results can be inconsistent. This can happen because of new insertions while a user is viewing a paginated page.
  
  ❌ For large set of data, pages in the tail end can take longer to be queried.
Keyset Pagination
- The client sends a keyset representing the last item in a page (instead of an offset and a limit). The next page is then defined as the records following that last item in sorted order.
- keyset is a set of columns that can be used to sort records in a stable order.
- Make sure an index exists for the columns used in the keyset
- DRF has it built-in, use CursorPagination!
- django-keyset-pagination is another third party library for this.
- Flaws
  
  ❌ Lack of tooling support in vanilla Django
  
  ❌ Can not select arbitrary pages, only the next page or the previous page.

I enjoyed the technical tidbits as much as I enjoyed the accompanying fantasy adventure. If you're interested to buy the book, head over to Spellbook Press website. Support indie publishers!

Cover Image Credit: Spellbook Press web page of the book

This article was originally published at my personal blog

Generate Generators Repeatedly

Modasser Billah — Mon, 15 Jun 2020 16:59:38 +0000

Generators are great. We all know it. They save us all the memory lists take up yet give us the same results and more. Well, almost.

The only catch with generators is that they generate the values only once. Let's look at an example.

def generate_numbers(num):
    while num > 0:
        yield num
        num -= 1

The function above gives us a generator. It takes a number as an argument and produces all the numbers from there down to 1.

generate5 = generate_numbers(5)

If we print generate5 it will show us that it holds a generator.

print(generate5)
>>> <generator object generate_numbers at 0x10dcb2b88>

Now we can do something like the following to generate the numbers:

for number in generate5:
    print(number)

And this will print:

But if you run it again

for number in generate5:
    print(number)

You won't see any output.

Well, this is what generators do. They generate values on the fly and then stop. You can't run them again and again unless you run generate5 = generate_numbers(5) each time. Generally people will tell you in cases like this you should use a list.

But there's a way around if you insist on using generators yet want repeated outputs. Let's see how.

class NumberGenerator:
    def __init__(self, num):
        self.num = num

    def __iter__(self):
        count = self.num
        while count > 0:
            yield count
            count -= 1

The code from the generate_numbers function is copied over (well, almost with slight class related modifications) to the __iter__ method for the NumberGenerator class. Whenever we loop through an object, it looks up the __iter__ method. If it's there, Python will use it to return an iterator.

So, now you can do something like this to instantiate the class:

generate5 = NumberGenerator(5)

And then loop over it as many times as you want using:

for number in generate5:
    print(number)

Note that this may not be the optimal thing to do all the time but it's an interesting tidbit to know that you can hack a generator this way to work more than once.

Happy hacking!

This post was originally published here

Crack the Code

Modasser Billah — Sun, 12 Apr 2020 17:56:32 +0000

With quarantines and lock-downs, most people around the world are stuck and bored in their houses. A lot of fun riddles are circulating in news feeds on social medias. Here's one of those.

Can you crack the code? It's fun to solve by hand but can you crack it using Python? Here's how I did it:

from itertools import product


def first_fifth_condition(candidate, condition):
    common_elements = [digit for digit in candidate if digit in condition]
    return (len(common_elements) == 1) and (
        candidate.index(common_elements[0]) != condition.index(common_elements[0])
    )


def second_condition(candidate):
    condition = (1, 8, 9)
    common_elements = [digit for digit in candidate if digit in condition]
    return (len(common_elements) == 1) and (
        candidate.index(common_elements[0]) == condition.index(common_elements[0])
    )


def third_condition(candidate):
    condition = (9, 6, 4)
    common_elements = [digit for digit in candidate if digit in condition]
    if len(common_elements) != 2:
        return False
    for digit in common_elements:
        if candidate.index(digit) == condition.index(digit):
            return False
    return True


def fourth_condition(candidate):
    condition = (5, 2, 3)
    common_elements = [digit for digit in candidate if digit in condition]
    return len(common_elements) == 0


def find_code():
    # for completeness test all possible permutations with repetition of digits for length 3.
    # We can further optimize by eliminating 0,5,2,3 from the possible candidate calculation.
    for candidate in product(range(10), repeat=3):
        if (
            first_fifth_condition(candidate, (1, 4, 7))
            and second_condition(candidate)
            and third_condition(candidate)
            and fourth_condition(candidate)
            and first_fifth_condition(candidate, (2, 8, 6))
        ):
            return "".join(map(str, candidate))
    return "No solution found."


if __name__ == "__main__":
    print(find_code())

Got better code to crack the code? Share links to your solution in the comments! Happy hacking!

Level Up Your Python Code Reviews

Modasser Billah — Sat, 11 Apr 2020 14:31:17 +0000

In the world of software development, there are many things that classes and books will prepare you for. But there are some topics that are often overlooked in learning resources yet hold immense value for the craft. Code review is a topic very high on that list. Knowing to review Python code well can land you into senior positions in no time.

In modern software development, code reviews play a pivotal role in ensuring the quality of software. This article takes a shot at explaining how to adopt Python philosophies to review code and then perform code reviews in a more Pythonic way.

At the end of the article, you'll be able to:

Understand the different aspects of Python code review
Develop a process to review code thoroughly and professionally
Detect areas of improvement in code with a higher probability

NOTE: This article assumes you know how to use git and how collaborative software development works using Pull Requests. If you're not familiar with git and GitHub, check out the Hello World GitHub Guide.

Let's get started!

Objectives of Code Review

Before you dive into how to do a good Python code review, let's first contemplate on why good code reviews are important. Peer review is the scientific process of verifying new knowledge. This is a widely practiced method in the scientific community.

Code review is the equivalent process in the world of software development. No one is perfect. That's the primary reason for code reviews. It always helps to have a fresh pair of eyes go through the code.

The benefits of code review can be broadly categorized in the following 3 categories:

Catching bugs
Ensuring code quality and maintainability
Building relationships

Let's explore how code reviews help us achieve these benefits.

Catching Bugs

Code review helps catch bugs before they go into production. Performance issues are also addressed in code reviews. This helps enhance the quality of the software in terms of both efficiency and correctness.

Ensuring Code Quality and Maintainability

Code is read much more often than it is written. Like in any engineering discipline, maintainability matters a lot in software. Code reviews ensure that someone has already read your code and understood it or raised the issue and helped you make it more readable and maintainable.

Building Relationships

Friendship happens on the way to something else. Think of all the friends you have in your life. You didn't become friends just for the sake of being friends. Rather you shared a pursuit with them and in the process became friends.

This is often overlooked but code reviews can spark conversations. Through collaboration, discussion and sometimes, the disagreements that happen on code reviews, you can often befriend fellow developers if you have the right mindset for code reviews.

Python Code Review: The Human Aspect

The human aspect is often the one overlooked in code reviews. Good technical reviews can lose effectiveness due to poor communication and lack of empathy. You should try to keep it human. Having a few pointers in mind helps in this regard. Let's try to establish some.

You're Reviewing Python Code Written by Humans

When reviewing a piece of Python code, it is important to factor in the human aspect. You need to make sure that your legitimate concerns and feedback are not nullified by your lack of empathy.

NOTE: People often make jokes about developers for lacking social skills which is not entirely true but has some truth to it. So, choose your words carefully and try to be positive. Be critical but courteous.

When you point out an issue, always try to explain why. It may be obvious to you and you may feel inclined to just make a one-liner comment to change this to that. But it may not be obvious to others including the author.

Explaining the why helps you to connect with the authors and gives them a chance to respond correctly. It also serves as a learning point for anyone going through the pull request.

The tone and warmth of the reviewer is lost in transmission on a written medium. As a result, comments with the best of intentions can be perceived as harsh. So, whenever possible, try to keep it warm with appropriate emojis to propagate your good vibes 🙂

However, the other extreme can also occur at times. Sometimes people overdo it to sound less critical and the message loses its meaning. Writing generic comments or always following a certain template can expose the mechanical nature of the review and make the author feel like you don't mean it.

There's No Exclusively Correct Opinion

While best practices do exist, there are very few binary rights and wrongs in coding. Building software is an art and like any other art, a lot of it is subjective. So, while you should not shy away from expressing and defending your opinion, remember that your opinion is not the last word on the topic.

There may be other perspectives, there maybe flaws in your argument and there may be more convincing argument behind the original proposal. So, keep your mind open while you express your opinion and don't hesitate to change your opinion once there's compelling evidence to do so.

Even if there's none, accept the fact that your opinion can be disregarded. Remember the phrase, strong opinions loosely held. You fight for your opinion but don't cling to it like revelation.

At times, developers tend to go overboard in the heat of the moment while defending an opinion. This reduces the objectivity of the discussion and makes the author take it personally.

One trick to avoid this is to make a context switch before you finally post your review. Let's say you wrote a long review and explained how it can be done better. Don't hit the comment button instantly.

Instead, take a moment to get your head out of the zone. Maybe check your mail or twitter feed for a moment or take a sip of the coffee waiting on your desk. Then come back and read your review. More often than not, you'll feel inclined to change a word or two, or rephrase a sentence. It may feel like a small thing but it can do wonders!

You may not know the author in person but you know for sure that it is written by a human. So, be nice and follow the golden rule, treat others the way you want to be treated.

Embrace Feedback Don't Dread It

On the side of the author, code review can be a stressful event, especially for people with less experience. It's easy to feel demoralized and think you don't know how to write code when someone requests some changes.

Imposter syndrome is real and everyone feels it. This can also make people defensive and rigid about accepting reviews. That's why it is important to keep in mind that the reviewer is trying to help.

There's nothing personal and no blame game going on. Objectively taking feedback is a great skill. Whenever it feels personal, remind yourself that the ultimate objective is to improve code quality.

TIP:
If a review is unclear or you don't understand the reason behind it, don't be afraid to ask why. As said before, coding is an artistic endeavor.

So, ask for reasons politely and present your arguments if you disagree with the reviewer. Learning is also an important outcome of the code review process. If you accept all requested changes without asking questions, you are depriving yourself of valuable learning opportunities.

Python Code Review: The Technical Aspect

This is the most critical part of the review. A review has to be technically sound to add value to the process of building softwares. You should have a rigorous process to sift through the code during a review. Keep iterating over the process to make your review a little better every time. Good news is, you can borrow from the collective wisdom of the developer community in this regard.

Stand on the Shoulder of Giants

Software engineering has a rich history. A lot of smart people have already figured out a lot of cool things to help you build better software. Be smart and don't reinvent the wheel. Rather, build from what's already in place. Let's dive into some of the first principles you can use all the time.

The Zen of Python

Python is much appreciated for the design philosophy behind it. You can build your Python code review muscles by starting from the guiding principles of the language. Much of this is beautifully condensed in The Zen of Python.

You can read the Zen of Python any time you want from a Python shell:

>>> import this

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Wisdom and first principles are often simple but not simplistic. That's why it's easy to overlook them. But sticking to them can really improve the quality of both the code and the review.

For example, you can often come around code like this:

if is_authenticated:
    return some_private_data()
else:
    return not_allowed()

But Zen of Python teaches you that flat is better than nested, so you can suggest this to be done this way:

if not is_authenticated:
    return not_allowed()

return some_private_data()

The example above is over simplified but you get the point.

Readability

Readability is often underestimated but it's very important for maintainability. It is a central philosophy that guided the development of Python as a language. Think about the ideas of indentation based scoping and no semicolons. Remember, the first consumer of your code is you. So make it readable for your future self.

When you're new to Python and learning a lot of cool tricks, it's easy to assume that writing complex one-liner code demonstrates that you're a good Python developer. Same can be said about reviewers, you can feel that you have suggested an improvement by recommending a one-liner in place of a code block. Look at the following snippet for example:

print("x is 3" if x == 3 else "x is not 3")

It's only one of line of code but takes the reader longer to understand what it does. Whereas you can just write:

if x == 3:
    print("x is 3")
else:
    print("x is not 3")

This is easier to read and understand but you may not feel like a genius when you write it. That's where you need to be more careful.

Code is communication. Computers only do maths and shuffle bits around. There are infinite ways to make a computer take a certain input and create a certain output.

The code you choose to use should mainly and most importantly tell other developers what it does. That's why Python is such a great programming language, it's readable. The Zen of Python backs it up as well.

Best Practices

Best practices exist for a reason. It helps you to benefit from the experiences of the people who figured things out before you. So, use them as guiding lights. If you see a single letter variable name while doing a Python code review, then ask why. Don't stop on why, suggest a better descriptive name.

Similarly, when you read a code block and understand what and how it does but not the why, ask why. Suggest that the why should be included as a code comment.

Tests

In a Python code review, it's important to ensure that the piece of code under review is well tested and documented. Tests ensure integrity of the code. If the Python code you're reviewing has new functions or classes added but no tests, then politely ask why. One hundred percent code coverage is not always feasible, so it's important to understand the author's perspective and suggest writing tests based on the case in hand.

There are different well accepted testing libraries like pytest and Python's own unittest library. It is better to choose one instead of mixing different approaches together.

Documentation

Tests are a good documentation of what the code does. But it is also important to document Python code, especially public APIs. That's why you should check functions, classes and methods for docstrings and suggest adding one if there's none.

There are a few accepted styles of documentation. For example, NumPy-style, Google Docstring convention and so on. Any of these should be good to follow but it is important to remain consistent throughout the project.

Style Guides

One other important resource that you can readily borrow from others and put to use is style guides. Style guides keep code organized. Remember, coding is akin to art and art is subjective. If an agreed upon style is not enforced, the codebase can become messy and hard to maintain.

PEP 8 is the guiding principle in this respect and you should refer back to it from time to time. Most IDEs come with linters. Linters check for style violations and display errors. Using a Python linter like Flake8 can help reduce a lot of style related errors. You can also use a formatter like black to handle formatting for you.

Nevertheless, it is essential to have a good grasp on the PEP 8 guidelines and look out for related issues when you're doing a Python code review. Flake8 Rules is a great resource to learn about it from examples.

Value Structure and Encourage Systems Thinking

So far, this article focused on how you can benefit from the collective wisdom of Python developers in your Python code reviews. But each project is different and strictly sticking to any guideline out of context can cause more troubles than it solves. That's where your situational awareness comes in.

You need to know the objectives of the project. The type of the project, the deadlines you need to meet, the people that will be using your product -- all should be factored in your overall strategy.

For example, if you're working at a start up and building a prototype, then functionality may be prioritized over maintainability. Code with less test coverage may be allowed to go through. In this case, your Python code review should be more focused on functionality and testing.

But if you're working on an enterprise solution, then you may need to focus more on maintainability and quality. Issues that could have been overlooked for a small project may not be allowed to go through in an enterprise solution. You may also need to be more rigorous in following up on style guides and internal coding standards.

Similarly, you may need to adapt your reviews depending on whether the product is in production already or not.

Be Aware of the Structure of the Project

This is where another Zen of Python principle comes into play, practicality over purity. You need to be pragmatic about your approach. Have a clear idea about which parts of the codebase can be more flexible and which parts need to be more airtight.

Disregarding principles is a bad idea and in most cases, special cases aren't special enough to break the rules(Yet another Zen of Python wisdom). But irrational adherence to them is not ideal either. So, keep the trade-off in mind when you suggest a change in your Python code review. Keep an eye out to ensure that the codebase is modular and well structured. One component should only do one thing. Otherwise, soon every module will have a dependency on every other module and the codebase will start looking like a big ball of mud.

Don't let perfect be the enemy of good. Sometimes it's better to get something merged and improve on that work in future pull requests.

Code in isolated components, modules, or packages often shouldn't be held back for minor issues. Instead of holding back the pull request, point out the area of improvements and approve the existing work on the condition that your points be considered during future iterations on this. Importantly, be sure that you always highlight these desired changes and follow up on them.

In general, you can be more lenient regarding the how of a system than the what. For example, what a public method does should be thoroughly inspected but the implementation of the method can be improved upon iteratively. This is because future improvements should only have effects within the system not with the outside world.

But know when to stand your ground. Contrary to the case of isolated components, shareable components should be held to a higher standard. Unclear naming or overly complicated interactions between parts of the system are a very real problem and will quickly become difficult to alter.

It's still important to avoid letting perfect be the enemy of good, but if significant changes will be difficult to perform later due to widespread usage, it's usually worth spending a little time on that early in the development process.

NOTE
Public code (for example, public interfaces, APIs) should be critically evaluated for ease of use and understanding before releasing because future changes may require changing every interaction throughout the system.

Often times, people come into a project and start suggesting changes that are Pythonic in principle but may not be in accordance with the existing codebase. No one and no codebase is ideal but you need to ensure local consistency before you suggest a change otherwise.

For example, a codebase may have been using single quote docstrings but you prefer the NumPy docstring style of double quotes. It is important to not introduce a change that creates confusion and parallel conventions in the codebase. Remember this from PEP 8:

"A Foolish Consistency is the Hobgoblin of Little Minds."

-- Guido van Rossum (source)

So, make sure you're locally consistent before trying to achieve a general consistency.

Leverage Systems Thinking

Systems thinking is the ability or skill to perform problem solving in complex systems. Software systems are inherently complex. In order to provide useful feedback, you need to be able to identify the constituent parts of the system and the interactions between them. Developing a mental model for systems thinking can help you detect areas of improvement in your Python code reviews. Think of code as a collection of systems and the interaction among them. This applies to reviews as well.

The whole is greater than the sum of the parts.

-- Aristotle

It's easy to get pulled into the code block under review and forget the bigger picture. So, it's important to step back and think about what the code does without getting into the how immediately. That's where the quote from Aristotle comes into play. The code may work fine in isolation but cause issues when it runs as a part of a system.

Reflect on how the feature could fail. Think about ways to break the code. Many a times, developers code for the happy path. That is, they write code in such a way that it only handles the best possible path.

But real world is messy. Users can do stupid things, machines can fail and cause errors. So, look out for scenarios that are likely to happen but not considered in the solution. Here's a very simple example:

def calculate_velocity(distance, time):
    return distance / time

The function above calculates velocity. It works well for most cases except when time is 0. So, it will throw an error in that case but the code doesn't handle that error.

This is bad and no foreseeable errors should go unhandled. So, thinking like a hacker can help you provide good insights in your Python code reviews.

Similarly, you should try to think of the problem without being influenced by the solution of the author. This helps to point out the assumptions made by the author in the solution. This conforms to another Zen of Python principle: In the face of ambiguity, refuse the temptation to guess.

For example, look at the following code block:

import random

def assign_random_numbers(users):
    result = {}
    for user in users:
        result[user] = random.randint(1, 10)
    return result

The function above assigns a random number to each user in the list of users passed in the argument. It creates a dictionary mapping between the users and a random number and returns the dictionary.

It may look like a good solution at first glance but it has two assumptions attached to it.

Let's look into the assumptions:

It uses user as the key for the dictionary. But for that to hold true, the elements in the users list must be hashable.
This solution assumes that all the user names in the passed users list argument will be unique. But names can be duplicate in real world and in case of a duplicate name, values for the same key would be overwritten.

NOTE
Developers often want to solve problems and impulsively make assumptions to go through the problem at hand. Be aware of this in your Python code reviews and you will be able to provide important feedbacks and prevent untested assumptions sneaking into production code.

Performance is also an important topic to review. Depending on the scope of the project, performance can be critical. Even when it's not so, it is good to not let non-performant code go through.

For example, if you see nested for loops, ask why the author wrote the code this way and suggest an alternative approach if one exists.

Using profiling to test the performance of the code may also uncover insights into performance issues. These may be more applicable to api responses and complex queries.

General Technical Considerations

There are some general issues to consider as well that are not necessarily related to Python but are important in terms of code reviews.

Keep the following in the back of your mind when doing a Python code review:

A Pull Request Should Do a Single Atomic Thing: Code reviews are mainly done on pull requests. So, the number of commits in a pull request or the scope of the pull request is important for review. Ideally, it should be a unit of functionality that can be tested in isolation. This makes reviews easier and reduces the probability of bugs sneaking into production. If you see a pull request with 30 commits and 50 files changed, then it is a good idea to suggest that the pull request be broken into several smaller pull requests.
Commit Messages Should be Relevant and Meaningful: Commit messages should be reviewed as well. Commit messages preserve the history of the project. So, if you see a lot of work in progress commits, then you can request squashing the commits to keep the history clean and coherent. If you see meaningless commit messages like bug fix or wip and so on, then you should ask to modify the messages to something more meaningful. It is good to know How to Write a Git Commit Message.
Typos and Grammatical Mistakes Should Not be Overlooked: Developers can code from all parts of the world and that means grammar issues in variable names, code comments, documentation and git commits are a common occurrence. Typos are frequent as well. It is important to point these out in a friendly manner but not hold back important features that need to get through.

It's a good idea to start your review with these items. That way, you get a good overview of the pull request under review and can dive deep into the details.

Python Code Review: The Organizational Aspect

Code reviews happen in a collaborative environment and successful collaboration needs careful organization. Setting up clearly defined processes around code review can speed up the development workflow and increase productivity and quality.

If you've read this far, then you may have already noticed that there is a pattern. It is possible to automate and standardize parts of the Python code review process to raise the level of productivity as a team.

Standards and guidelines can help align all the team members to produce coherent code and reviews. You can further raise the bar of reviews by incorporating checks like continuous integration, code coverage, linters and so on.

For example, style guides are mostly static. So, style checks can be automated using bots like Lintly, GitHub actions or pre-commit hooks. These checks will automate parts of the things you need to review and allow you to focus on the code more.

Developing a pull request description template is also very useful for Python code reviews. A description template helps the author to present his pull request in a systematic fashion. It also helps the reviewer to learn the overall scope of the pull request before delving into the code. An example pull request template can be like this:

## Problem
Small description of the problem

## Accepted Solution
A small summary of the approach taken to solve the problem

## Instructions to Test
Some step by step instructions or automated commands to test out the solution

## Links to related work
Links to dependent pull requests, user stories, bug reports,
support tickets and so on

Creating a pull request template for your repository is very helpful. This way, everyone creating PRs on the repo will automatically get the template in their PR body to fill in the sections.

You have been introduced to a number of aspects to consider during your Python code review. Remembering all of this can be hard initially. Using checklists for reviews can help you approach your Python code review tasks more systematically.

You can prepare a personal checklist to go through during your reviews. Here's an example of what it can look like:

There are tests for new code being added.
There are no typos, grammar issues.
Code is readable and variables and functions have meaningful names.
There is documentation for public methods.
The solution handles errors properly.
Overall, the code conforms to the Zen of Python and best practices.

These aspects are organizational because these make the most impact when adopted across the whole team. Knowing about this also provides you an opportunity to suggest organizational processes to the team. This can optimize overall Python code review workflow and enable you to contribute more to the team.

Conclusion

Hopefully, you've come across a lot of new concepts in this article. Learning and applying it well can help you become a better reviewer and in the process a better technical leader.

To summarize, this article expanded on how to:

Analyze the process of Python code review
Approach each part of the process accordingly
Look back at the whole process to optimize it for you and the team

It is easy to get bogged down into the details and lose focus on the bigger picture. So, let's revisit the proper scope for Python code reviews one last time. Gratitude multiplies, so does frustration. So, be generous, kind and empathetic in your reviews.

Don't overcomplicate things, rather keep it simple and encourage others to write more Pythonic code. Be receptive to feedback when you're on the receiving end and consciously iterate on feedback to improve. Lastly, remember that the goal is to come out of the Python code review process a bit better, for all the 3 entities involved: the author, the reviewer and the codebase. Now, go perform Python code reviews like a pro!

Acknowledgements: I have drawn a lot of inspirations for this article from my time at BriteCore. Some ideas are directly from Lennart Regebro and Daniel and Audrey Feldroy. Thanks to all my colleagues there, past and present, who have put in place wonderful processes and standards for code review.

The article was originally published here

Shell Function for Pulling PRs

Modasser Billah — Thu, 18 Apr 2019 09:18:51 +0000

Fetching PRs opened by others for reviewing or testing new functionalities is a common task. Generally, you'd do the following for this:

git fetch upstream pull/<PR_NUMBER>/head:<BRANCH> && git checkout <BRANCH>

Typing this everytime can soon become irritating and you can't have an alias for it because PR number and branch name will vary.

So, shell function comes to the rescue! Functions can take parameters, so it makes our lives easier in this case.

So, if you want a shorthand for the above commands, try the following:

Put this in your shell configuration file like, .bashrc or .zshrc :

Source the config file with source ~/<Shell_Config_File_Name>
Fetch and checkout a PR branch with: pr <PR_NUMBER> <USER:BRANCH>

The shell function is a bit opinionated. I prefer to name branches as originally named by the author in their fork. This helps keep track when you have a lot of branches.

GitHub has recently introduced a copy button beside the branch name.

So, it makes it easier just to click that button to copy the branch name but it comes with a user name as well. But you don't have to worry about it, just copy it as it is with the button and use that as the second variable to call the shell function. The first line of the function splits and takes only the branch name to use it as the new branch name.

Happy hacking!

Initially published at: http://modasserbillah.ml