DEV Community: Nikhil Taneja

Screen to Screen Navigation Understanding with AI

Nikhil Taneja — Thu, 17 Jul 2025 19:29:03 +0000

Current AI agents for automated UI testing have an expensive problem: they make LLM calls at every single step to decide what to do next. Most agents cache simple input-output pairs, but they don't build any understanding of the overall application structure. This leads to high API costs, slow execution, and inefficient navigation.

What if we could flip this approach? Instead of real-time decision making, what if we pre-computed the entire app's navigation structure from manual testing recordings? Could this comprehensive knowledge base provide better caching than simple input-output storage? Could it reduce the cost of agentic app testing and make navigation faster? A tool that not only maps every screen connection but also understands the context and user intent behind each transition?

That's exactly what I built using Microsoft's Omniparser and Google's Gemini LLM. I tested it by analyzing the complete navigation flow of the Rapido customer app, but the approach can be applied to any website, desktop, or mobile application.

Processing Pipeline

1. Video Frame Extraction

Endpoint: POST /extract_video_frames/

Intelligent video processing that automatically identifies key moments:

Algorithm Details:

Frame Sampling: Processes at most 5 FPS regardless of video FPS
Change Detection: Uses histogram correlation and MSE on downsampled frames
Quality Preservation: Outputs full-resolution frames despite optimized processing

Configuration:

{
  "sensitivity": 0.3, // 0.1-1.0 (lower = more sensitive)
  "min_time_gap": 0.5 // Minimum seconds between extractions
}

2. Frame Management

# Get extracted frames
frames = requests.get(f'/extractions/{extraction_id}/frames/').json()

# Select frames for processing
requests.post(f'/extractions/{extraction_id}/select_frames/', json={
  "frame_ids": [1, 3, 5, 7, 9],
  "action": "keep"  # or "delete"
})

3. UI Element Detection with OmniParser-v2

OmniParser is a screen parsing tool to convert general GUI screen to structured elements like icons, text and their bounding boxes.

# Process selected frames with Omniparser
requests.post(f'/extractions/{extraction_id}/process_selected/', json={
  "batch_size": 4
})

4. Screen Analysis with Gemini

The system uses a sophisticated prompt template that analyzes each screen across 7 dimensions:

Screen Identification: Type, purpose, and user journey stage
Visual Analysis: Layout, colors, accessibility features
Interactive Elements: Function and expected outcomes for each element
Screen Transitions: Possible navigation paths and conditions
Context & State: Required data and error states
Knowledge Base Entry: Structured data for machine consumption
Questions & Uncertainties: Areas needing clarification

Endpoint: POST /extractions/{id}/knowledge_base/

Example Analysis Output:

### 1. SCREEN IDENTIFICATION

- **Screen Type**: registration_final_step
- **Screen Purpose**: User completing profile setup
- **User Journey Stage**: End of onboarding flow

### 6. KNOWLEDGE BASE ENTRY

SCREEN: registration_final_step
CONTEXT: User completing profile setup in onboarding
ELEMENTS: name_input, gender_selection, whatsapp_toggle, next_button
TRANSITIONS:

- tap: next_button → main_dashboard (condition: name_filled AND gender_selected)
- swipe: back_gesture → previous_registration_step
  AUTO_TRANSITIONS:
- form_completion → welcome_screen (timing: immediate)
  NOTES: Final onboarding step, critical for user activation

5. AI-Powered Cleanup

Endpoint: POST /extractions/{id}/knowledge_base/cleanup/

Used Reasoning model to:

Standardize naming: Convert to snake_case conventions
Deduplicate screens: Identify and merge similar screens
Fix transitions: Map target screens to correct standardized names
Enhance quality: Add missing context and fix inconsistencies

6. Final Data Structure

Data Structure:

[
  {
    "screen": "booking_pickup_selection",
    "context": "User selecting pickup location for ride booking",
    "elements": [
      "search_input",
      "map_view",
      "location_suggestions",
      "confirm_button"
    ],
    "transitions": [
      {
        "trigger": "tap",
        "element": "confirm_button",
        "target": "booking_destination_selection",
        "condition": "location_selected"
      }
    ],
    "auto_transitions": [
      {
        "trigger": "location_detected",
        "target": "booking_confirmation",
        "timing": "3_seconds"
      }
    ],
    "notes": "Critical step in booking flow"
  },
  {
    // ... other screens
  }
]

Graph-Based Path Discovery

The system constructs a complete navigation graph from knowledge base entry and uses pathfinding algorithms to discover all possible routes between any two screens.

Each path includes interactive screenshots, transition metadata, and step-by-step navigation instructions.

Building Navigation Intelligence for Autonomous Systems

Original Intent: Knowledge Base for AI Agents

The initial vision was ambitious: create a comprehensive knowledge base that AI agents could use to understand and navigate websites and mobile applications autonomously. The idea was that with detailed screen-by-screen analysis and transition mapping, AI agents could plan complex multi-step navigation sequences to accomplish user goals.

The Theory: An AI agent could query the navigation knowledge base to understand:

"What screens exist in this app?"
"How do I get from the home screen to the checkout?"
"What are all the possible paths to cancel a booking?"
"What elements should I interact with to complete a purchase?"

The Reality: AI Agents and Immediate Action Bias

However, during testing with various AI agents, an interesting pattern emerged. AI agents consistently focused on immediate, first-step actions rather than using the comprehensive navigation knowledge for long-term planning.

When presented with the complete navigation graph and detailed pathfinding capabilities, AI agents would:

✅ Use screen context to understand the current state
✅ Identify interactive elements on the present screen
✅ Determine the next logical step in a workflow
❌ Rarely plan multi-step sequences using the full path knowledge
❌ Seldom leverage alternative routes when primary paths failed
❌ Infrequently use the pathfinding for complex goal achievement

The sophisticated pathfinding and navigation intelligence that seems so valuable for human analysts may be less immediately useful for AI agents than originally envisioned.

Future Uncertainty: Where Does This Lead?

This disconnect between the tool's capabilities and AI agent usage patterns leaves me uncertain about its future direction.

Results and Implications

After testing with the Rapido customer app (200+ screens):

Technical Achievements:

200+ screens analyzed from user session recordings
500+ navigation transitions mapped automatically
Sub-millisecond pathfinding between any two screens

Self-Hosting Applications on Raspberry Pi 5

Nikhil Taneja — Sat, 17 Aug 2024 09:54:00 +0000

Introduction

Self-hosting applications can be a rewarding experience, providing full control over your software environment and data. In this blog post, I'll share my journey of setting up a self-hosted environment using a Raspberry Pi 5, detailing the hardware, software, and techniques I use to maintain and monitor my system.

Hardware Setup

Raspberry Pi 5 Specifications

CPU: Quad-core Cortex-A76 at 2.5GHz
RAM: 8GB LPDDR4
Storage: 1TB SSD attached via Pimoroni PCIe HAT
Power Supply: Official Raspberry Pi charger, with inverter power backup for up to a couple of hours
Network: The Raspberry Pi is connected via LAN to a router with an FTTH connection from the ISP
Operating System: Debian based Raspberry Pi OS Lite

Software Setup

Security and Access

I use Cloudflare Zero Trust for a secure web gateway. Each application is exposed on a different subdomain. Access to these applications is restricted via the Warp client, with Cloudflare also offering the ability to create API token-based access. Cloudflare provides extensive logging and monthly overviews of network usage.

Application Deployment

Applications are deployed as docker containers running in rootless mode. Containers are managed using Dockge via stack-oriented docker compose.yaml files.

I also utilize self-hosted GitHub Actions runners to automate CI/CD for my own repositories.

Favourite application

Immich

Immich is a self-hosted photo and video backup software that aims to replicate the convenience of Google Photos without the recurring costs. Immich has been a game-changer for me, especially during my recent trip to New York. With my iPhone storage filled up from taking hundreds of photos in a single day, Immich effortlessly allowed me to upload and organize my images, freeing up space almost instantly. It has all the features I expect like smart search, facial recognition, media transcoding and sharing along with intuitive interface and reliable performance made managing my photo storage a breeze, ensuring that I could keep capturing moments without any hassle.

Monitoring and Logging

For accessing Docker container logs, I use Dozzle, a simple and effective tool for debugging.

My monitoring stack includes Prometheus, Grafana, Alertmanager, and various exporters like Node Exporter, cAdvisor, PostgreSQL Exporter, Cloudflared, and services exposing OpenTelemetry metrics. These metrics are visualized using community-built Grafana dashboards.

System monitoring
Database monitoring
Gateway monitoring
HTTP Server monitoring

Critical alerts, such as high CPU usage or frequent 5xx errors, are sent to me via Slack through Alertmanager rules.

Challenges and Solutions

Loose PCIe SSD HAT cable

I experienced SSH connectivity issues with random spikes in ping latency and sometimes even a 'Host is down' message when trying to ping my server. Making me think that it is network related issue. I tried both LAN and WiFi connections but still the problem persisted.



// ping stable for small duration
64 bytes from IP ADDR: icmp_seq=0 ttl=64 time=6.467 ms
64 bytes from IP ADDR: icmp_seq=1 ttl=64 time=2.575 ms
64 bytes from IP ADDR: icmp_seq=2 ttl=64 time=3.246 ms

// ping latency spikes after some duration
64 bytes from IP ADDR: icmp_seq=3 ttl=64 time=4992 ms
64 bytes from IP ADDR: icmp_seq=4 ttl=64 time=14331 ms
64 bytes from IP ADDR: icmp_seq=5 ttl=64 time=17592 ms

// followed by timeouts
Request timeout for icmp_seq 6
Request timeout for icmp_seq 7
Request timeout for icmp_seq 8
Request timeout for icmp_seq 9

// eventually connection disruption
ping: sendto: No route to host
Request timeout for icmp_seq 10
ping: sendto: Host is down

I also noticed that the SSD drive wasn't detected using the lsblk command, which led me to suspect a power-related issue. I thought my charger might not be supplying enough watts to handle the Pi 5, active cooler, and SSD. So, I purchased an official Raspberry Pi charger, but the problem persisted.

I had both SSD as well as SD Card mounted with OS installed on both of them. Boot priority was set to give preference to the SSD over the SD card. However, the system would sometimes boot from the SSD and other times from the SD card. Occasionally, the system would run from the SSD for a few hours, but then it would automatically restart and switch to the SD card. During travel, the PCIe cable connecting the Pi to the HAT became loose. Whenever this system reboots, Host becomes unavailable explaining why I faced network interruption. This also explains why the issue started occurring after moving to Bangalore. Debugging this was quite challenging.

Solution:



Make sure all hardware connections are secured

Rootless containers exit once the user session exits

Running Docker in rootless mode caused containers to stop after exiting the SSH session. I have a habit of not closing SSH connection opened in the terminal. Even if I close my laptop (just lid closed, not a shutdown), the SSH connection still remains open for small duration. After that duration user logs out and all containers exit making my applications unaccessible.

If establish SSH connection, it would bring all containers back online.
If keep my laptop open, containers kept running forever.

Solution:



loginctl enable-linger $UID

Random Shutdowns

My Raspberry Pi experienced random shutdowns, which I linked to CPU IO spikes through monitoring.

dmesg logs indicated that the NVMe SSD was entering power-saving mode, causing these shutdowns.



[  603.860958] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
[  603.860967] nvme nvme0: Does your device have a faulty power saving mode enabled?
[  603.860969] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off" and report a bug
[  603.956075] nvme 0000:06:00.0: Unable to change power state from D3cold to D0, device inaccessible
[  603.956325] nvme nvme0: Removing after probe failure status: -19
[  603.970331] nvme0n1: detected capacity change from 1953525168 to 0

Solution:

Edit /boot/firmware/cmdline.txt to disable APST



sudo vim /boot/firmware/cmdline.txt

Add the following in the end



nvme_core.default_ps_max_latency_us=0

Reboot and see if you have disabled APST.



sudo nvme error-log /dev/nvme0

Too many open files

Initially, Applications performed well, but over time, their performance degraded. The root cause was low default ulimits. I resolved this by increasing ulimits. I also added a fan cooler to maintain optimal temperatures to avoid thermal throttling.

Solution:

Edit /etc/security/limits.conf to set limits for users or groups



sudo nano /etc/security/limits.conf

Add the following lines to set the limits:



*    soft core            unlimited
*    hard core            unlimited
*    soft data            unlimited
*    hard data            unlimited
*    soft priority        0
*    hard priority        0
*    soft fsize           unlimited
*    hard fsize           unlimited
*    soft sigpending      63980
*    hard sigpending      63980
*    soft memlock         8192
*    hard memlock         8192
*    soft rss             unlimited
*    hard rss             unlimited
*    soft nofile          1048576
*    hard nofile          1048576
*    soft msgqueue        819200
*    hard msgqueue        819200
*    soft rtprio          0
*    hard rtprio          0
*    soft stack           8192
*    hard stack           8192
*    soft cpu             unlimited
*    hard cpu             unlimited
*    soft nproc           unlimited
*    hard nproc           unlimited
*    soft as              unlimited
*    hard as              unlimited
*    soft locks           unlimited
*    hard locks           unlimited

Reboot / Reload the Configuration



sudo reboot

Verify the Changes



ulimit -a

This will display the current limits applied to your session.

Conclusion

Overall, I'm quite happy with my setup. It allows me to quickly self-host applications. However, I still need to devise a robust backup strategy. Currently, I manually back up data to a physical hard drive, which is not ideal. I also plan to explore scaling storage and compute, potentially setting up distributed systems with separate nodes for databases and applications to meet future needs.

Final Thoughts

Self-hosting applications has been a valuable learning experience. It has equipped me with the skills and confidence to manage my infrastructure, troubleshoot issues, and optimize performance. I look forward to further enhancing my setup and sharing more insights with you all.

Happy self-hosting!

Done is better than perfect

Nikhil Taneja — Sun, 19 Dec 2021 15:13:24 +0000

Introduction

"Done is better than perfect" is a lesson that I learned from working in a fast-paced competitive startup environment.. and agile software development in general.

Every new codebase I used to touch, I could find some place where something can be improved by using a different data structure, or an algorithm, or some design pattern avoiding nested if-else, or simply abstracting big functions into smaller ones. But as when I started working on big features I came to realize this famous saying.

In this article, I would like to share some of my thoughts on this ideology and hopefully give you some motivation to release your next amazing startup or project you are working on and not get yourself stuck in catch-22.

Perfection

the state or quality of being perfect

In software development world - When you’ve developed a working product, it’s normal to take pride in its performance you’ve worked hard on it. It’s normal to want to hold off on release until it’s got this feature, or that’s been tweaked, or it’s gone through yet another round of testing.

Source: The ‘done is better than perfect’ approach to programming - Parker Software

Chasing perfection

If somebody asks me if the system I have developed is perfect or not? I would say comeback after couple of years and ask the same question as I cannot answer it today. Gone are the days where software was sold via floppy disk/DVDs, now software's are getting more personalized and unique to each and every customer. System once perfect needs to evolve with new requirements which one would have never anticipated while designing/developing it.

Over-engineering is often identified with design changes that increase a factor of safety, add functionality, or overcome perceived design flaws that most users would accept. It can be desirable when safety or performance is critical (e.g. in aerospace vehicles and luxury road vehicles), or when extremely broad functionality is required (e.g. diagnostic and medical tools, power users of products).

As a design philosophy, it is the opposite of the minimalist ethos of "less is more" (or: “worse is better”) and a disobedience of the KISS principle. - Wikipedia

Example

Let's say you are a student and as an assignment your teacher asks you to submit a research report on sustainable future or solar energy before next week. If you submit your assignment after the deadline, your marks will get affected even though how in-depth you went into research analyzing sun's orientation, affect of weather, calculating perfect tilt angle, finding how much power output one solar panel can generate, finding return on investment... doing justice to the topic.

Even if your teacher was generous and because of your amazing report gave you full marks. This is not always the case, imagine if this was a question part of your final exam and if you spend all the time answering it so well, you will not get enough time to answer other questions.

Some might argue in the above example a short-crisp, to the point answer would have been perfect. In reality it takes a lot of good efforts, right knowledge and experience, to setup a great foundation and even then you have to adapt to changes and keep on enhancing the product.

Maybe this example is not perfect, so are the projects we work on. There are memes around project requirement not being clear enough. Not everyone is building a medical device where margin of error is a difference of life and death where you have to consider using special tools like ROS to control system clock and scheduler. Not every requirement is like The Thames Barrier must never fail. Here's why it doesn't. - YouTube

Finding the balance

Understand the requirements

Let's say you are working on a feature that will be very important for winter vacation season sale - Chrisman and New Year. Then this is a strict requirement which you have to follow. Christmas is always on 25th of dec. and if your feature is not ready by then business can get affected. That being said not all requirements are strict and it is important to understand them well to avoid doing unnecessary work. Writing good readable code, following coding standards (linters) and writing testcases should be part of requirements of a good tech team and should be caught in code reviews.

Once I was almost about to write a wrapper library on top of RestAPIs of one of our web service. Was it part of the project requirements? - No! Were estimates taken in account for writing the library? - No!

Know your customer

Your customer is the consumer of the features you build. You should be aware of the impact it can have on them not just positively but repercussion if anything goes wrong. This provides sense of responsibility you have, skipping on testcases and not doing non-functional requirements testing reduces your confidence so it is advised to given them attention. Trust me, it makes you happy seeing your customers are happy.

Will customer's care about the technology used? - No/maybe! Will customers have a bad experience if they are not able to login? - Definitely!

What are your expertise

One cannot have knowledge of everything. You should know your boundaries and capabilities. Technology evolves rapidly and it's very hard to keep up-to date with everything. This not only restricted to knowledge of a particular programming-language/framework.

If you are asked to make changes in a totally new codebase, You are no longer an expert and it's your responsibility to get enough context/knowledge from the right person to do full justice to the requirements and communicate this thing clearly to your manager.

Engineering process is a loop

We(Software engineers) have luxury to ship the Minimal Viable Product (MVP) and constantly evolve it incrementally. We do not need to know the final state of the project in the beginning. We should be able to collect necessary feedback and act upon data what works and what does not. This also applies to issues, gives us ability to accept stop gaps while actual fix gets released in next sprint.

Good read: Scaling with common sense - Zerodha Tech Blog

ESLint rule: react/jsx-curly-brace-presence

Nikhil Taneja — Fri, 03 Sep 2021 19:05:39 +0000

I like to read engineering and technology related articles shared by wonderful folks our various platform like Medium, Dev.to, Twitter, etc. One day while scrolling through my feed I stumbled upon #bugsmash program.
I looked into the open issues and picked Add eslint rule for formatting redundant curly braces in jsx #12643. During this time I was reading about ASTs, learning how linters work so I thought it would be a good issue to tackle.

About the bug

Add eslint rule for formatting redundant curly braces in jsx #12643

aitchiss posted on Feb 10, 2021

This issue is to consider enabling the rule for react/jsx-curly-brace-presence in our eslint config. This could be set up to catch two formatting/code style issues, e.g.

<MyComponent type={ 'exampleType' }>{ 'Hello world' }</MyComponent>

Would be corrected to:

<MyComponent type="exampleType">Hello world</MyComponent>

It's a minor code style issue, but would make our JSX components more consistent and readable. We already have eslint-plugin-react as a dev dependency in the project, and could enable this rule as a warning initially.

NB: From a bit of investigation it seems Prettier can't do this for us which is why I'm suggesting eslint

View on GitHub

The task was to be set up ESLint rule to catch the following formatting/code style issues in react



<MyComponent type={ 'exampleType' }>{ 'Hello world' }</MyComponent>

Would be corrected to:



<MyComponent type="exampleType">Hello world</MyComponent>

As, you can see below, I ran the linter on all JavaScript files and it found and fixed 1 issue here ->

How I Smashed It

#12643 Add eslint rule for formatting redundant curly braces in jsx #14592

itsnikhil posted on Aug 25, 2021

What type of PR is this? (check all applicable)

[x] Refactor
[ ] Feature
[ ] Bug Fix
[ ] Optimization
[ ] Documentation Update

Description

Initially I thought I would have to build and publish my very own eslint plugin to solve this issue but later on I found out that such linting rule already exists. After merging this PR you can expect to see this linting check in action fixing where

<MyComponent type={ 'exampleType' }>{ 'Hello world' }</MyComponent>

Would be corrected to:

<MyComponent type="exampleType">Hello world</MyComponent>

As, you can see below, I ran the linter on all JavaScript files and it found and fixed 1 issue here -> https://github.com/itsnikhil/forem/commit/17508f12970621a323711cc5f08e0a67b7a7977b#diff-55437c4c12bdc7f18b73ceb37af6c27f95e647ef5ada26083c7fa53bd3730832R112

Related Tickets & Documents

This PR fixes https://github.com/forem/forem/issues/12643

QA Instructions, Screenshots, Recordings

You can run the following command and such linting violations will be marked as errors

 yarn run lint:frontend

And with --fix, it will try to automatically fix such errors

 yarn run lint:frontend --fix

UI accessibility concerns?

N/A

Added/updated tests?

[ ] Yes
[x] No, there are no unit tests for linting violations.. I have done manual testing but if there is some way please let me know.

[Forem core team only] How will this change be communicated?

Will this PR introduce a change that impacts Forem members or creators, the development process, or any of our internal teams? If so, please note how you will share this change with the people who need to know about it.

[ ] I've updated the Developer Docs and/or Admin Guide, or Storybook (for Crayons components)
[ ] I've updated the README or added inline documentation
[ ] I've added an entry to CHANGELOG.md
[ ] I will share this change in a Changelog or in a forem.dev post
[ ] I will share this change internally with the appropriate teams
[ ] I'm not sure how best to communicate this change and need help
[ ] This change does not need to be communicated, and this is why not: please replace this line with details on why this change doesn't need to be shared

What gif best describes this PR or how it makes you feel?

View on GitHub

The solution was as easy as adding any other existing rule. Since eslint-plugin-react was already added as a dev dependency in the project, I followed the guide here https://github.com/yannickcr/eslint-plugin-react/blob/master/docs/rules/jsx-curly-brace-presence.md



// .eslintrc.js
rules: {
    /* 
        Existing rules
    */
    'react/jsx-curly-brace-presence': [
      'error',
      { props: 'never', children: 'never' },
    ],
  },

Before reaching to above solution, I explored https://astexplorer.net/ and how to work with rules. I also learned how to write my very own ESLint rule.



<MyComponent type={ 'exampleType' }>{ 'Hello world' }</MyComponent>

The above jsx in AST have following structure:

So, we can have a following function which apply on all JSXExpressionContainer using visitor pattern and check if typeof node.expression.value is a string then we can replace the expression with a fixer by replacing it with node.expression.raw



function (context) {
    return {
      JSXExpressionContainer(node) {
        if (typeof node.expression.value === "string") {
          context.report({
            node,
            message: "Do not use redundant curly braces",
            suggest: [
              {
                desc: "Replace `{ 'string' }` with `'string'`",
                fix: function (fixer) {
                  return [fixer.replaceText(node, node.expression.raw)];
                },
              },
            ],
          });
        }
      },
    };
  }

You can see the whole thing in action
https://astexplorer.net/#/gist/b146b6ba734c554f51e4d5bb53023dda/e4077cfc126335c9468cca9eba01c675addf177d

Reflection

It was great experience building forem's source code and I like that the DEV team really appreciated my PR however small it was.

Architecting discord bot the right way

Nikhil Taneja — Fri, 27 Aug 2021 14:01:14 +0000

Past: Gmbot

If you might not know, I have made a discord bot Gmbot - Multiplayer Game bot for Discord (itsnikhil.github.io). The way how this bot used to function was very simple, there was a trigger phrase gmbot followed by command name like createwar and some parameters

gmbot createwar -players @Player1 @Player2 -game 4 -duration 12 -type ft3

I have already written a detailed post showcasing it's various features, database schema, etc. which you can read at Gmbot - Chatbot for gamers (itsnikhil.codes). I made that bot in python as discord.py API wrapper library was written in python and I was comfortable working in that language. Like everyone, I followed whatever tutorials, documentation, etc. were available online. But today in this post I want to talk about something which I did not give much thought at the time when I started working on gmbot in 2019.

Everything was very monolithic

What I meant with monolithic is that all the functionality existed in a single codebase, everything I wanted to do had to be done in python. Even though the project was well structured into classes, easy to read/understand code and a common pattern was followed for creating each command but it was not testable, not scalable and not flexible. All the business logic was mixed within the command handler.

Example of how "gmbot status" command looked like

import discord
import DAO

class Status_Command(commands.Cog):
    def __init__(self, client):
        self.client = client

    @commands.command(aliases=['wars'])
    async def status(self, ctx, *args): # status command handler
        # validate command
        # call DAO to get data from database
        # business logic
        # call success/error
        if message:
            await self.success_message(ctx, message)
        else:
            await self.error_message(ctx, "ERROR")

    async def success_message(self, ctx, error):
        # format and send success message

    async def error_message(self, ctx, reason=None):
        # format and send error message

def setup(client):
    client.add_cog(Status_Command(client))

Can we do better? Micro-services

By micro-services I do not mean having different bots handling different features of the bot. I want you to think and treat discord bot server as simply a frontend-client like a mobile app and instead of mixing business logic into it, have a separate backend server communicating via RESTful APIs.

gmbot createwar -players @Player1 @Player2 -game 4 -duration 12 -type ft3

With this architecture commands like above will be passed to chatbot frontend-client which can then make a POST request to to backend with the params in the body and client key in headers indicating this is a record from discord.

curl --location --request POST 'localhost:8080/createwar' \
--header 'client: DISCORD-BFF' \
--header 'x-api-key: **************' \
--header 'Content-Type: application/json' \
--data-raw '{
    "server": {
        "id": "S1",
        "name": "Server1 Name"
    },
    "players": [
        {
            "id": "P1",
            "name": "Player1 Name",
            "role": "member"
        },
        {
            "id": "P2",
            "name": "Player2 Name",
            "role": "co-leader"
        }
    ],
    "game": "Mini Miltitia",
    "duration": "12h",
    "format": "first-to-3-wins"
}'

This API would return JSON response and status code based on which frontend-client can craft a discord formatted response or slack formatted response

embed = discord.Embed(
    title="WAR REQUEST ACCEPTED",
    url="https://itsnikhil.github.io/gmbot-site/commands/",
    description="Your clanwar request has been accepted...",
    color=0xffff00
)
embed.set_author(
    name="GMBOT",
    url="https://itsnikhil.github.io/gmbot-site/commands/",
    icon_url=self.client.user.avatar_url
)
for key, value in response["data"]:
    embed.add_field(
        name=key,
        value=value,
        inline=False
    )
embed.set_footer(text="Clanwar accepted!")

# Inform host that clanwar request has been accepted
await info_channel.send(embed=embed)

Advantages

Much easier to test the features
Easy and quick growth/expandability
Backend servers can be horizontally scaled
Flexibility to experiment and optimize
Freedom to try out new technology
Separate deployments
Leaner teams and increased productivity

Much easier to test the features

We have been developing backend services for many years now and have developed complex observability tools which can even tell which API request took how much time for a particular database query in real-time. Of course we have proper tooling to test and automate backend servers with proper CI/CD process. Without separate backend there is no good way to test discord/slack bots besides testing manually. If you would search for testing in discord.py official docs Search (discordpy.readthedocs.io) you will be out of luck.

Easy and quick growth/expandability

If you create a chatbot for discord, you have to accept the fact that you are limiting your audience group to that matching discord's. Sketch design tool restricted itself to only support MacOS while Figma was web based open-to-all because of which Figma easily captured the market however good Sketch was in it's comparison. Let's say you developed an attendance bot and you want to support both slack and teams, backend can still be shared while clients can vary since core business logic is same across bots/ webapp. This is something I regret not doing for gmbot, when gmbot failed, I could not pivot my idea into a mobile app easily. Having this opportunity is very important for a lean startup which can require pivoting business into different direction any day.

Backend servers can be horizontally scaled

Let's say your chatbot got famous, everyone is talking about it, your product is in the headlines everywhere <3. There will be a flood of new users trying to use your service. According to some random folks on internet, 1 server can handle ~2500 active discord groups load. This can obviously vary based on your server resources and what actions your bot perform. Fortunately, discord.js bot support sharding Getting started | Discord.js Guide (discordjs.guide) but I would argue, it is much better to scale backend horizontally by running multiple instances of server behind application load balancers, using in-memory caches, etc. these are webservers in the end, you don't have to depend on discord to provide a way to scale.

Flexibility to experiment and optimize

Adding/Removing features becomes much easier, you can setup multiple environments like production and development allowing you to safely work on experimental features without affecting actual customers or their data. You can also work on optimizations behind the scenes without touching the frontend-client

Freedom to try out new technology

Freedom to use different technologies without getting locked to any single programming language. Business logic can be written in golang with it's high performance benefits from compiled binary and goroutines while still using discord's JavaScript library in frontend-client to communicate with discord. If something better comes along, you have the ability to try it out. You can have different parts of the service handled by different microservices.

Separate deployments

Each microservice has separate codebase which can be updated independently of other services. Microservices plays really well with continuous integration and continuous delivery by providing ability to have different deployment pipelines.

Leaner teams and increased productivity

Micro-service codebases are smaller in size and thus much faster to build, test and release. One do not depend on other to make a release after which you can release yours. We can have people work on domains/areas where they have expertise. Not being forced to collaborate with everyone let's developers focus on their tasks and increase productivity.

Clarify function calls with keyword arguments - JavaScript

Nikhil Taneja — Sat, 07 Aug 2021 12:34:24 +0000

While working with many JavaScript libraries you might have come across this common pattern where in the function call you pass an object many-a-times often referred as options. There is a reason why this is a common good practice and in this article I will be giving the reasoning behind this practice.

Common options pattern

Let's say you are working on an existing codebase and see the following code

twitter_search('@obama', False, 20, True)

Some eagle eyed person amongst you might have noticed I have F and T capital in Boolean, that's because it's not JavaScript. This example had been taken from a talk "Transforming Code into Beautiful, Idiomatic Python" by Raymond Hettinger.

In a company there are many people working on same codebase and let's say it was not you who wrote the above code. You might be able to tell this function will search for tweets where Obama has been tagged and maybe you could guess we need to fetch 20 such tweets but False and True what does that mean? You would have to memorize the arguments to check that.

In Python, solving this problem is simple using named arguments. In this way, your function call would become

twitter_search('@obama', retweets=False, numtweets=20, popular=True)

But unfortunately we do not have luxury of named parameters in JavaScript but we have objects to the rescue.

twitterSearch('@obama', {
    retweets: false,
    numtweets: 20,
    popular: true
});

If you have worked with mongoose library, they share the similar API

await mongoose.connect('mongodb://localhost/my_database', {
    useNewUrlParser: true,
    useUnifiedTopology: true,
    useFindAndModify: false,
    useCreateIndex: true
});

From the library developer perspective

Let's say you are writing a module which formats date into a specified format (just for the sake of an example)

const formatToString = (day, month, year, format, sep) => {
    const dd = day.toString().padStart(2,0);
    const mm = month.toString().padStart(2,0);
    const yyyy = year.toString();

    switch (format) {
        case 'dmy' : return `${dd}${sep}${mm}${sep}${yyyy}`;
        case 'mdy' : return `${mm}${sep}${dd}${sep}${yyyy}`;
        case 'ymd' : return `${yyyy}${sep}${mm}${sep}${dd}`;
        default : return `${yyyy}${sep}${mm}${sep}${dd}`;
    }
}

module.exports = {
    formatDate(date, format, separator, options={skipInvalid: false, overrideInvalidWith: new Date()}) {
        options.skipInvalid = options.hasOwnProperty('skipInvalid') ? options.skipInvalid : false;
        options.overrideInvalidWith = options.hasOwnProperty('overrideInvalidWith') ? options.overrideInvalidWith : new Date();

        const dateObj = new Date(date);
        if (isNaN(dateObj)){
            if (options.skipInvalid)
                return null;
            let overrideInvalidValue = new Date(options.overrideInvalidWith);
            if (isNaN(overrideInvalidValue))
                overrideInvalidValue = new Date();
            dateObj = overrideInvalidValue;
        }
        const day = dateObj.getDate();
        const month = dateObj.getMonth() + 1;
        const year = dateObj.getFullYear();

        return formatToString(day, month, year, format, separator);
    }
}

This module export formatDate public function which take date, format, separator and a couple of optional parameters. And the function call will be like

formatDate("2021-08-07T12:06:07.484Z", "ymd", "-") // without options
formatDate("2021-08-07T12:06:07.484Z", "ymd", "-", {skipInvalid: true}) // converts invalid date to null
formatDate("2021-08-07T12:06:07.484Z", "ymd", "-", {overrideInvalidWith: "2021-08-07T12:06:07.484Z"})

// special edge case which need to be handled by library properly using hasOwnProperty
formatDate("2021-08-07T12:06:07.484Z", "ymd", "-", {})

Checking if options have been initialized properly with hasOwnProperty might slow down your code by a little bit but really what are you trying to save "micro-seconds" or "hours of programmer's time"? I hope your answer is hours of programmer time. This simple transformation improves code readability a lot.

Another advantage of having options is that we can add other optional arguments with sensible defaults without breaking existing functionality from the library consumer's end.

General rule

Whenever you see a need for having a optional parameter which has some default value, consider using options object as one of the parameter which will provide all these benefits. For private function like formatToString in the above example it does not make use of options object because it has not been exposed to the outside world, it's scope is limited to that particular file only.

Understanding concurrency in go

Nikhil Taneja — Tue, 03 Aug 2021 19:17:21 +0000

Concurrency is about dealing with lots of things at once. Parallelism is about doing lots of things at once - Rob Pike

Go language itself provides some features to handle concurrency out of the box, hiding all the complexities so that developers write better, faster, more efficient code. These features include :-

Goroutines: A goroutine is a lightweight thread managed by the Go runtime.

Just add go in front of your function call to convert it into goroutine and you can take advantage of concurrency.

func main() {
    go say("hello") // concurrently executes func say
}

// you might also see the following pattern
func main() {
    go func(greeting string){ // concurrently executes anonymous func and 
        say(greeting) // call our function inside it
    }("hello")
}

Channels: Channels are a typed conduit through which you can send and receive values with the channel operator, <-.

ch := make(chan int) // unbuffered channel
ch := make(chan int, 30) // buffered channel
ch := make(<-chan int) // Receive only channel
ch := make(chan<- int) // Send only channel

ch <- 1 // sends value to channel
x = <-ch // assign value from channel to x

close(ch) // close channel and clean memory

WaitGroups: A WaitGroup waits for a collection of goroutines to finish.

var wg sync.WaitGroup

func main() {
    wg.Add(1) // Add to set the number of goroutines to wait for
    go func(asyncFuncParam string) {
        // each of the goroutines runs and calls Done when finished
        defer wg.Done() // executed after functions returns
        asyncFunc(asyncFuncParam)
    }("USING WAIT GROUP")
    wg.Wait() // block until all goroutines have finished
}

Concurrency in go

If you are not taking advantage of goroutines then ask yourself why not? I mean why don't you want your code to go faaassstttt? While there are a lot of cases where sequential execution of code is important, here we really cannot do anything but there are some situations where we could take advantage of concurrency but uncontrolled concurrency is harmful. For example: Making multiple concurrent API requests might get you in trouble of getting rate limited HTTP 429 - Too Many Requests. Luck you, even in such cases we can take advantage of language features like channels and use it's properties to limit concurrency.

In this post, I have shared 2 different ways of handling concurrency

Uncontrolled concurrency

This will queue as many goroutines to execute in concurrent mode as your system can handle. Use this when your program has many autonomous pieces independent of each other

var wg sync.WaitGroup

for _, item := range items {
    wg.Add(1)
    go func(asyncFuncParam string) {
        defer wg.Done()
        asyncFunc(asyncFuncParam)
    }("USING WAIT GROUP")
}
wg.Wait()

Controlled concurrency

The will create a bounded queue and limit goroutines according to a set limit per seconds while execute in concurrent mode following bucketing/ short bursts pattern. Use this when your program has many autonomous pieces independent of each other but you are rate limited due to some bottleneck.

const (
    CONCURRENCY_LIMIT = 30 // max 30 items in a channel per second 
)

guard := make(chan struct{}, CONCURRENCY_LIMIT)

for _, item := range items {
    guard <- struct{}{}
    go func(asyncFuncParam string) {
        defer func() { <-guard }()
        asyncFunc(asyncFuncParam)
    }("USING CHANNELS FOR CONTROLLED CONCURRENCY")
}

Benchmarks

200+ API requests

Mode	Time of Execution	Outcome
Synchronous	~1 min 20 sec	All API succeeded
Uncontrolled concurrency	~5 sec	Many API failed with error
Controlled concurrency	~10 sec	All API succeeded

No doubt uncontrolled concurrency will be the fastest to complete the job but in the end it failed to get all the response successfully. With controlled concurrency, I can manually tweak performance and find right balance between rate limits and execution time.

Learning resources

Concurrency section of A Tour of Go - One of the best official resource https://tour.golang.org/concurrency/1
If you want know internal workings of goroutines, i would highly recommend you read the following article https://medium.com/the-polyglot-programmer/what-are-goroutines-and-how-do-they-actually-work-f2a734f6f991

Implementing silent refresh of JWT

Nikhil Taneja — Sun, 30 May 2021 22:19:27 +0000

Prerequisite knowledge

When you are building services for everyone to access online, you need have a way protect resources so that only the user can access data belonging to them only. For this, websites implement login/ signup, enter username and password and voila access granted! This process is called "Authentication". But what happens when you close your browser or refresh the website. Do you re-login with username and password? No, right. To solve this there is one more thing called "Authorization" which is actually making sure that the user which sends requests to your server is the same user who actually logged in during the authentication process. There are a lot of strategies like Session IDs, API key, Token auth, etc. which comes with the problem of storing client secret securely at a secure place.

Session IDs

Session ID is a random secret id generated by the server which is stored in database having one to one relationship with primary key of a particular user. This session id is sent to client whenever you login via cookie. Whenever you make a request to private resource. Browser automatically adds this cookie in the new request which is validated by the backend. This system works but can there be a better solution where we do not need to maintain a state or a database record? Enter JWT

JSON Web Token (JWT)

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

JSON Web Token (JWT) token follows similar flow diagram to Session like after authentication instead of sending session id in cookie we send a JWT token and for other subsequent requests, this token is used to authorize the user. The key difference here is instead of storing session ids in database we store all the necessary user's info in the token itself. JWT token consists of 3 parts:

Each separated by period (dot) and encoded in Base64. Payload is the JSON object containing this data. This token is then stored in the client either in cookie or on browser's localStorage. JWT tokens are signed with a secret key (generally HMACSecret key) following digital signature cryptography, it invalidates the token when tampered by the malicious user.

JWTs are "stateless", which unlocks some benefits like:

No re-login required for separate services, as same JWT token can be used to authorize as long as all services share same signing secret key. The data is stored in client JWT's payload and not in a particular service's database.
The token are stored on the client, that's really the import thing about JWT so no matter if there is load balancer or multiple micro services, the user can authenticate with any of those servers as long as they share same secret key.

But with stateless nature, we do not have any mechanism to logout/ stop the access of the user. Typically this token is dropped from localStorage or cookie from their browser but we are trusting the client here, another approach can be to implement a blacklist of restricted tokens which essentially defeats the purpose of stateless. Also, client storing token on localStorage can be vulnerable to XSS attacks or CSRF attack if JWT is stored in cookies.

All this leads to the gist of this post, following a better approach of having 2 tokens - one short lived access token and another long lived refresh token.

Introduction to short lived token

Understanding the aspects of access tokens and refresh tokens can be a bit tricky. Whenever a user "authenticate", server sends 2 tokens - access token and refresh token to the client. The speciality of having 2 tokens is they we have expiry on these tokens as part of JWT's payload. Expiry duration of access token is significantly shorter of ~10mins to 24 hours than that of refresh tokens ~months to years or even no expiry. When user want to access any private resource they require access token to "authorize" and everything else works just like with a single token. We can generate new access tokens from refresh tokens.

Where to store the token?

So now the question comes what is the benefit of having access token if we can generate it from refresh token.

LocalStorage

Storing JWT tokens in local storage can leak the token in an event of an XSS attack as hacker can run arbitrary JavaScript code which can provide them access to in-memory variables and localStorage

Cookies

Storing token's in cookie can lead to CSRF attack which can make user perform a private action without knowing (or knowing when it's too late) like phishing user to perform /account/delete

What is the best way then? According to hasura's ultimate guide "access token" should be stored in "memory" and "refresh token" should be stored in a "secure HTTP only cookie". CSRF wouldn't work as access token will not be added automatically and even if hacker can get user to generate new access token via CSRF, they cannot read the access token from response. Now, if XSS is found hacker would still be able to read access token from memory but since the token is short lived the threat level is reduced.

Memory

In-memory!! What if user refresh the webpage or re-open the browser? access token will not be available!

In such cases, since refresh token would still be there in cookie we can use it to perform a silent refresh of access token.

Server's database

Some companies do store refresh token or a unique id of as salt of refresh token (called refresh secret) in database. I am personally against this ideology as it is against JWT's stateless nature but sometimes there can be requirement to logout user immediately. Even in such scenarios find out exact scenarios when you would want to forcefully logout user. Like when user's password is changed or when we suspend a user's account we want their access to terminate immediately. In such cases why not use isSuspended and passwordHash as part of your refresh salt instead of storing random uuid as refresh secret for each user in database.

Silent Refresh

Silent refresh is a mechanism to generate new access token from refresh token automatically in the event of browser refresh or when access token is expired but refresh token is available and valid. According to Hasura's guide this is handled at the client side. Client makes very first request to /refresh_token as the website loads. This approach is handles browser refresh event but for when token is expired we have to have a setTimeout event and make another API call to do silent refresh.

The BAD way

Handling silent token refresh on sever side, the way I am sharing below is NOT a good idea. Refreshing token automatically via middleware will render it susceptible to CSRF attacks as we are only depending on refresh tokens which is stored in HTTP only cookie. So, it seems like best approach is the one suggested by Hasura only, it might force you to make a GET request to /refresh_token and handle logic of refreshing token on the client side which at the moment is safest way!

On server, having a middleware IsLoggedIn whose purpose is to validate the JWT and authorize access. Now, instead of just checking for access token, it also look for refresh token when access token is missing or invalid. IsLoggedIn middleware can generate new access token from the details in token's payload and return the new-access-token in response header. Client can implement an interceptor to update value of in-memory access token.

From the image above I want you to notice 2 things:

Success response on a private resource without providing access token (aka simulating browser refresh)
Getting new x-access-token in headers using refresh token which was provided via cookie.

The Hasura's (GOOD) way

Scenario 1: Page refresh

Since we are only storing access token in memory when page is refreshed we lose access token. This can be handled my making very first requests to /refresh_token and generate a new access token. This silent refresh will have 2 steps as explained in the below diagram.

Scenario 2: Access token expired

Since access token is short-lived, it will expire very soon and requests will start failing. Whenever this happens client would need to make a request to refresh token again and then retry same request with the new access token. Alternatively, an event can be scheduled to run on every setInterval to refresh access token or some even adopt a strategy to returns new access tokens on every valid private resource request.

References:

JSON Web Tokens - jwt.io

https://www.youtube.com/watch?v=iD49_NIQ-R4

https://hasura.io/blog/best-practices-of-using-jwt-with-graphql

This had been a great learning lesson for me which I would have not learned if I used any 3rd party library.

Algorithm behind TNB Analysis

Nikhil Taneja — Mon, 25 Jan 2021 11:25:09 +0000

TNB or thenewboston is a digital cryptocurrency network which is being developed by Bucky Roberts and open source community of which I am part of. There are thousands of registered accounts which will increase rapidly in future. At the time of writing this article there are 90 JSON files. My task was to process each of these JSON files and output results providing insights like total coins distribution over time period, total number of accounts, richest account information, wealth distribution, etc. Now that you know background, problem statement and assence of the scale of challenge let's see how I solved it.

Through this article you will learn about JSON Streams and Sorted set empowering the algorithm behind TNB Analysis project

Datasource

Everyday for about past 3 months, Bucky or someone from the team takes Account model backup as JSON and push it to GitHub which you can find account backups here. The data is a big object of objects. Opening it on mobile browser hangs my phone lol

{
"6649dde16e": {
    "balance": 166700,
    "balance_lock": "c8fa0404d9"
  },
  {...},
  ...
}

In the above example, "6649dde16e" is a particular account's public address. Balance is yes you guessed it - TNB coins a particular person owns. You can ignore balance_lock for now as it is irrelevant for analysis.

Processing JSON files efficiently

Task was to read each object in the JSON one by one and sum up balance for each account, easy right?

const backup = require("path/to/backup/file.json");
let totalBalance = 0;
for (let account in backup) totalBalance += backup[account].balance;
console.log(totalBalance);

But this is not the best way of doing things for a particular reason. When it comes to reading a large array of objects (for example, 10,000 objects), we are usually required to deal with major performance issues regarding large memory allocation for objects. Nodejs has a default memory limit of 512mb on 32 bit and 1 gb on 64 bit systems. To deal with the issues, we can use Streaming only small chunks of valid json. I used stream-json npm package.

const { chain } = require('stream-chain');

const { parser } = require('stream-json');
const { streamObject } = require('stream-json/streamers/StreamObject');

const fs = require('fs');
const pipeline = chain([
fs.createReadStream(file),
    parser(),
    streamObject()
]);

let total = 0;
let max_balance = 0;
let rich_account = '';
let n_accounts = 0;

// Process each account object in json
pipeline.on('data', (data) => {
    ++n_accounts;
    total += data.value.balance;
    if (data.value.balance > max_balance) { 
        max_balance = data.value.balance;
        rich_account = data.key;
    };
});

// Calculate and print totals on completion
pipeline.on('end', () => {
    console.log(`Total: ${total} coins (${n_accounts} accounts)
    Richest account: ${rich_account} (${max_balance} coins)`);
});

Calculating wealth metrics

Now you can process whole JSON using streams in a single pass with very little memory footprint. Next challenge is to generate a rich list. Rich list is basically ordering account objects based on their balance. For this we keep accounts in memory. But do we need all the accounts? The answer is NO. At this point so far we already know total balance and total no. of accounts. So, if we want to find top 5% richest accounts then we should not care about rest of the 95% whose wealth will be total balance - 5%'s total balance.

I used "Sorted sets" data structure to achieve this. js-sorted-set internally uses Array or Binary tree or Red-black trees. Time complexity for each:

Operation	Array	Binary tree	Red-black tree
Create	O(1)	O(1)	O(1)
Length	O(1)	O(1)	O(1)
Clear	O(1)	O(n) (in garbage collector)	O(n) (in garbage collector)
Insert	O(n) (often slow)	O(n) (often slow)	O(lg n) (fast)
Remove	O(n) (often slow)	O(n) (often slow)	O(lg n) (fast)
Iterate	O(n) (fast)	O(n) (slowest)	O(n) (slower than Array)
Find, Test	O(lg n) (fastest)	O(n) (slowest)	O(lg n) (slower than Array)

I used red-black implementation of sorted sets mainly because of it's lg(n) complexity for insertion as that was my primary use case. Total time complexity for calculating rich list became (5% of totalAccounts) * lg(totalAccount) which is better than using merge sort or quick source algorithm along with the benefit of not keeping whole object in memory.

After calculating all these metrics, I save the results into CSV which is then used to generate charts using Pandas.

Thank you for your time. If you find the article insightful consider sharing it with your friends. TNB Analysis is open sourced https://github.com/itsnikhil/tnb-analysis and a star to project would be awesome.

Is pythonic way the better way?

Nikhil Taneja — Mon, 11 Jan 2021 20:22:44 +0000

Here I have shared some of my thoughts on some of the pythonic way coding style. These do not design patterns (which are best practices used by experienced object-oriented software developers) of any sort, just some ways of writing code differently or I should say more pythonic.

If you might not already know - Python is an interpreted, interactive, object-oriented programming language. But more importantly, Python is a programming language that lets you work more quickly and integrate your systems more effectively.

If you want to learn all the recommended python style conventions/guidelines, you can learn here. So with that, let's start with something which you might already know.

Swapping of values

This is a cool way of swapping values among two variables. Python does not create any temporary variables. It is all done on the stack.

a, b = b, a

This tuple unpacking not only makes code more readable but also handles the state better.

List comprehension

Most people working with python at a certain point of time discover about list comprehensions which are shorthand syntax of writing for a loop.

a = [i**2 for i in range(10)]

Awesome right, short and simple. If you use round brackets () instead of square brackets [] instead of returning a completely new list, it returns a generator object. Generators in python are super cool, they yields one value at a time which requires less memory. Any python function with a keyword "yield" may be called a generator. When the generator encounters a yield keyword the state of the function is frozen and all the variables are stored in memory until the generator is called again.

>>> a = (i**2 for i in range(10))
>>> print(a)
<generator object <genexpr> at 0x0000025CADCFA518>
>>> print(sum(a))
285

This is where the shortcomings of pythonic ways are prevalent amongst beginners, most beginners often use square brackets even when not required.

Compound comparison statements

A compound statement consists of one or more 'clauses'. An example for this will be

>>> x = 5
>>> 10 < x < 20
False
>>> x = 15
>>> 10 < x < 20
True
>>> x = 25
>>> 10 < x < 20
False

A compound comparison is a neat feature in python. In other languages, you would need to express this as two different comparisons joined with an and operation

>>> x = 15
>>> (10 < x) and (x < 20)
True

Distinguishing multiple exit points in loops

This is sort of like a replacement for flag variables. There are two ways you can exit out of the for loop 1) You hit the break; or 2) You did not. In the second case, else block will get executed.

>>> def find(seq, target):
...     for i, value in enumerate(seq):
...         if value == target:
...             break
...     else:
...         return -1
...     return i
... 
>>> find([3,4,5], 5)
1
>>> find([3,4,5], 6)
-1

I do not think there is any reason not to use this except the poor naming of else. I have not seen this used in many codebases maybe because it has a very niche use case, unlike list comprehension.

Ternary operator

Blueprint of general syntax

(if_test_is_false, if_test_is_true)[test]

Example usage

a = ('Odd', 'Even')[num//2 == 0]

This sounds a lot useful and you wonder why you might not have heard of this before or seen something like this in some codebase. There is a reason behind this, to understand consider this example

>>> (print('isFalse'), print('isTrue'))[True]
isFalse
isTrue

As you can see it prints both cases. Because of this not only extra processing is wasted but it can introduce some bugs. Ternary if else cannot be used to handle None value (or null value for non-python world).

>>> import re
>>> message = 'Is pythonic way the better way?'
>>> match = re.search(r'python', message)
>>> (match.group(), 'Not found')[match == None]
'python'
>>> match = re.search(r'python3', message)
>>> (match.group(), 'Not found')[match == None]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'group'

We received this error because when re.search fails, it returns None and we called a method on the something which was None. This happens because with the tupled ternary technique, the tuple is first built, then an index is found as a result both True case along with False gets executed.
So use this with a lot of care because the way I see this, it not a replacemrnt for if-else!

Walrus operator available from Python 3.8 (PEP 572)

Walrus operator can be used to consolidate an assignment statement and a boolean expression when both assignment and expression would utilize a similar statement.

>>> import re
>>> message = 'Is pythonic way the better way?'
>>> if(match := re.search(r'python', message)):
...     print(match.group())
... 
python
>>> if(match := re.search(r'python3', message)):
...     print(match.group()) # No output
...

Parentheses are important as without that it would assign result of boolean expression instead of the statement.

>>> my_list = [1,2,3,4,5]
>>> if (n := len(my_list)) > 3:
...     print(n)
...
5
>>> if n := len(my_list) > 3: # without parenthesis
... print(n)
...
True

Some good reads

This article has been inspired by a talk from PyCon US 2013 by Raymond Hettinger go watch it here
Mind-bending edge cases in python that make you say "Wat‽" go watch it here
There is a fun project attempting to explain what exactly is happening under the hood for some counter-intuitive snippets and lesser-known features in Python which you can read here

Thank you!

I hope you find this article insightful, please share it with your friends, you can read my other articles at my blog

If I have missed something, please write a comment and I will update my article.