DEV Community: Ninad Mhatre

Vibe coding - does it actually work?

Ninad Mhatre — Mon, 29 Dec 2025 09:40:45 +0000

I have been coding for 19+ years and for most of the time my workflow was pretty much similar. Things changed about 3 years back once ChatGPT came on the scene. The way I code has changed a lot after that. Still, at least for me it is far away from replacing developers.

What has changed?

Getting unstuck

In my company, AI tools are encouraged and Co-pilot and Windsurf are the official tools. Earlier (before 2022), If I was stuck on coding task, my go to option were "stack overflow" or asking a colleague. This was slow and most of the time, I had to post the idea without actual code and partially with limited context required for understanding the problem and then, wait till someone responds and further delays in case of follow ups if initial solution does not work.

This entire flow has changed to inline chat, AI has lot more context about my code and lot more knowledge about the libraries that my code is using. It often times give a solution and if not that, It gives me good pointers to solution. I sometimes switch between models to get code that suite my taste.

Of course, If there are up-sides then there has to be some downsides as well. AI also has downsides like, code generated with non-existing APIs, way to complicated code or code that does not work at all. For me, I always try to understand the code generated and change it as per my style.

The most important point here is that, ability to move forward rather quickly.

Bouncing Ideas

I remember once I wanted opinion of other to compare two ideas. I posted on stack overflow and question was banned as I was asking for opinion and not the definite answer. The whole point of question was to get opinionated answers but I also understand this could lead arguments between conflicting ideas.

With Copilot and Windsurf, I can bounce my ideas very easily and go deep to understand the solution. I can even ask for what references were used and follow them. I can ask it to generate quick code snippet to try out the idea and data required. All this without judgement and delays.

There is a good chance that you will get completely crap answers but its up to me to go deep and understand the suggestion and try it out before assuming that everything that is said is correct.

For e.g, I was working on Pandas based code and It was pulling data greater than 40 GB and then processing that much of data was clearly not best to be done with Pandas. I wanted to see if new "Polars" or existing "PySpark" was best for this task. And how much time will it take for computation in new approaches? I needed a POC to try out both ideas and see the result myself. Thanks for Claude I could complete the POC within few hours.

Auto complete & Testing

This has improved a lot, not only for code completion but for adding doc strings in code base. My code is now properly documented with example code.

Even before AI Tools I always added doc strings and examples but that whole process took longer and now, its auto-generated correctly in 95% cases.

One thing I am positively surprised about generated code is, sometimes as a developer I tend to use known API of any library instead of using new or better API which was added recently. I came across few PySpark & Pandas API that I learned because of AI tools.

I prefer to have partial code with test suite rather than all features and no test coverage. I try to build tests incrementally, and AI tools have helped me greatly to get started with test setup and adding basic tests. I iterate over basic tests and improve them, most of the time I keep the scaffolding created by tool and write my tests.

Learning

Learning is way easier with LLMs. Learn a new programming language or a library/tool (Terraform, Plumi, K8S) is very easy. You have inline help to get unstuck and learn quickly. You can ask all sort of questions without feeling being judged. All you need is curiosity.

I intentionally keep my auto-complete off when I am learning a language otherwise LLMs are so good that you will have a working code but zero-learning. Its a double edged sword!

I have realized that focusing on basics or concepts is far more important than learning a language or a specific tool.

Is it good enough to replace Devs?

I am impressed with the code generation ability of LLMs, but is it going to replace Devs? Short answer, I don't think so!

I have tried spec driven development when I wanted to build a job monitor, which would scan multiple database tables and compare the rows inserted everyday for last 7 days and flag if latest results are not in line with last 7 days.

I only used prompts to implement whole logic and for any changes relied on prompts. This whole process worked and I had monitor job created as per my wish in within 2 hours (accumulated all changes over a week).

One more example would be, I wanted to build a FastAPI based job scheduler, I (LLMs) built the UI for job scheduler using only prompts. I was 100% sure that I am not going to improve this code and just use forever once it works. Just like external package, as long as its working I am good!

If everything is working, then what is down side?

My Job monitor code generated 10+ files with 1000+ LOC and it explained me all the changed it did every time I asked it to change things. But that's just too much of code to review. It changed 500+ lines in HTML file, and I was asked for review those changes!

I prefer to understand the logic when I review instead of just looking for syntactical accuracy.

After so many years being a developer, One thing I know is, amount of code we maintain over time is far more compared to time we spend writing new code. I would say 80% is maintain and improve and rest is adding completely new code. I am just not sure how LLMs will handle this? Till now I have only seen LLMs producing new code but nothing about maintaining.

When I say maintain, I mean keeping the backward compatibility, thinking about the deployment, run time constraints etc. And adding new functionality in code base.

I feel confident when I know in-and-outs of code Bases I work on. And I don't mean line by line but the bigger picture, like which part deals with DB, file system or specific front-end component. In LLM generated code, I don't feel that confidence, maybe I need to invest more time to study the generated code to get that confidence.

Do you have a successful code base (with users) which is completely written and maintained by LLMs? Please let me know in comments. I would love to understand how its used.

Rust: cargo release flags or codegen options

Ninad Mhatre — Thu, 29 May 2025 09:39:07 +0000

I am learning Rust actively since a last 6 months, In order to learn I am building small personal projects to learn aspects of language. As I mentioned, I am still a newbie in Rust, so when Microsoft released edit (vim like editor) for windows which is fully written in Rust, I decided to take a look at the code base (I love open-source) in order to learn from it.

First thing I looked at was Cargo.toml and I was surprised by optimizations done for release build.

I decided to try those on my personal project. I created this small utility to parse parquet files using Rust (https://github.com/ninadmhatre/rupy/tree/main/pq). It works but problem is the binary size. Its massive 56 MB! I wanted to see how much reduction can be done on the binary size with this optimization/flags.

Default

Compilation Time: 2m 10s
Binary Size (MB): 56

Reduce codegen-units

For faster compilation code units are split, default is 256 (incremental) and 16. Higher the number quick the build but may produce slower code. Lets set it 1.

Docs: Here

Flags:

codegen-units=1

Compilation Time: 3m 10s
Binary Size (MB): 40

Change lto

Defers LTO optimizations to linker. Default is no or off.

Docs: Here

Flags:

codegen-units=1
lto= true

Compilation Time: 5m 16s
Binary Size (MB): 38

Change opt-level

Controls optimization levels. Default is no optiomization.

Docs: Here

Flags:

codegen-units=1
lto= true
opt-level = "s" // optimize for size

Compilation Time: 4m 35s
Binary Size (MB): 28

Rest of the flags

Below flags are about debug and controlling behaviour, so instead of checking 1 by 1, I applied them together.

panic: abort or unwind on panic?
strip: Controlles stripping of debuginfo and other data from binary.
split-debuginfo: emitting debuginfo into separate file

Flags:

codegen-units=1
lto= true
opt-level = "s" // optimize for size
panic=abort // do not unwind on panic
strip="symbols" // if you want crash reporting, then not recommended
split-debuginfo="packed"

Compilation Time: 3m 56s
Binary Size (MB): 20

My repository is very small but depends on Polars. In conclusion, with codegen options/release flags, my compilation time increased to 3m 56s from 2m 10s -> +81% but my binary size reduced from 56M -> 20M -> -65%. You can play around with these options to figure out what works for you.

You can read more about all codegen options

Python to Rust - Some early thoughts

Ninad Mhatre — Thu, 01 May 2025 08:14:05 +0000

I have been following Rust for very long time, I tried to learn Rust at least 3 times in the past 5 years. Last time I started learning rust, v1.36 was just released.

I always started to learn from the online rust book (which is very nice), In last 3 attempts I could only reach till chapter 9. This time i.e. about 8 months back, I decided to learn Rust completely and slowly. So I got myself a Udemy course (pre-recorded) and just followed it. I can say, I know basic and can do non-vibe coding.

Now to the topic of the post, but, before starting the post, a couple of quick disclaimers

I used Python for 10+ years and Rust for very short time, so this is based on very limited Rust knowledge.
Python is 30+ years old vs. Rust is ~10 year old, So new things often seem shiny!

1. Which packaging tool do you use?

In Python, there are multiple packaging tools. You have, basic pip to latest uv, and in between you have poetry, hatch etc.

I am personally comfortable with poetry but it's personal preference. As a team, It's not easy to agree on one tool when you have many options.

On top of this, you need formatting, linting, test runners. Agreeing on tools sometimes can be daunting.

It's hard to justify, in a 40K LOC project, which formatter to use ruff or black, unless you care about sub-second delay with black.

In Rust, there is only one, Cargo! It does everything. (For good or bad)

2. Libraries and separation of concern

In Python, I love pathlib library. It does everything, you can create path and then manipulate it.

import pathlib as pl

tmp_file = pl.Path("/tmp/this/is/new/dir/a.txt")
tmp_file.is_file()   # False
tmp_file.parents.mkdir(parents=True)

tmp_file.open("w").write("this is test file")

# How do you delete it?
# maybe, tmp_file.parents.rmdir() # nope, no option to delete dir-tree

With one module,

I created directory structure
I created file and wrote text in it.
It made it so simple.

But, then in order to delete tree you need another module. also inverse of mkdir, i.e, rmdir they don't work in similar manner.

import shutil

shutil.rmtree(tmp_file.parent)

In Rust, I did the same exercise and it required 2 different modules.

use std::fs;  // file-system
use std::path::Path;

fn main() {
    let path = Path::new("/tmp/this/is/new/dir/b.txt");
    println!("{:?}", path.is_file());  // same as python

    /* 
    Creating file and dir is handled by fs module (it can create and delete entire dir tree
    */

    fs::create_dir_all(path.parent().unwrap()).unwrap();
    fs::write(path, "this is a test file").unwrap();

    // fs::remove_dir_all(path.parent().unwrap()).unwrap()
}

Oh boy, but if you want to append to a file in Rust, it is not intuitive at all. also, I am not using fs::write to append to a file.

let mut append = fs::OpenOptions::new().append(true).open(path).unwrap();

append.write_all(b"\n and this is another line").unwrap();

3. Another example of library

Python, In order put a thread to sleep/pause it, you would do

import time

time.sleep(3)

I never questioned "why time lib that deals with time, is putting a thread to sleep?", until I did it in Rust

Rust, In order put a thread to sleep, you would do

use std::thread::sleep;
use std::time::Duration;

fn main() {
   sleep(Duration::from_sec(3));  // you have from_millis/micro/nano API to sleep for even shorter duration.   
}

As thread will be put to sleep, sleep is in thread module. and you have to use Duration object from time to specify how long to sleep?

4. import this

In Python, you have Zen of python, Its a philosophy that you can follow to write code. It was written in 1999!

You look at the current python ecosystem and wonder if those are even followed during python development. Lets look at them,

Note: I agree with those, just that they are sometimes hard to follow, also, since they are no concrete they are open to interpretation.

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

I have some use cases where I wonder,

Case 1

Lets look at,

tmp_file = pathlib.Path("/tmp").joinpath("a.txt")  # /tmp/a.txt

# but same library also allows this,
tmp_file = pathlib.Path("/tmp") / "a.txt"          # /tmp/a.txt

Its achieved by overridding __truediv__, good example of polymorphism, but is it "explicit"?

Breaks:

Explicit is better than implicit.
Readability counts.
Special cases aren't special enough to break the rules.

Case 2


set([1, 2, 3]) > set([3, 4, 5])  # is same as <set1>.is_subset(<set2>)

Why is this allowed? Some AI tool generated this code with "greater than operator", which I could only understand by reading docs. If readability counts why have such options built in the language?

Breaks:

Explicit is better than implicit.
Readability counts.

Case 3

The following examples exist mostly for legacy reasons, but they are still part of the language.

1. os.path.joinpath or pl.Path? 
2. glob.glob or pl.Path.glob?
3. 
   - "a %s", val 
   - "a {}".format(val) 
   - f"a {val}"?

Having multiple ways of doing same thing?

Breaks:

There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.

I don't have anything related to Rust like this, maybe due to my limited exp. or language is new compared to Python.

Now, coming to purpose of this post, I might never have questioned some of the implementation in my favorite language if I never tried same thing in another language.

It's good idea to learn a new language with a different philosophy every few years.

Python Caching mutable values

Ninad Mhatre — Sun, 26 Jan 2025 07:36:31 +0000

Caching is used everywhere in order to speed up processing. Its used right from CPU level to a layer in front of databases. Cache invalidation or when to remove something from cache, is very interesting and complex problem to solve. I am going to talk about something simple than that.

Issue am mentioning in this post is very simple, yet it took almost 1.5 years to come to the surface. It was hidden because of how the other parts of the code were written. It came to surface as more users started using they tried something apart from recommended approach. I will explain what it was later in the post.

Some background, I created custom ML Framework (sklearn based) within my current organization. It's a ML framework so as you guessed it, there are multiple data sources which are accessed frequently. I decided to add a layer of cache to speed up things, initially I started with lru_cache inbuilt module. Soon, I realized that we need some persistent caching as we fetch same data over and over (atleast while development) and all of that data is static data.

This layer of caching was needed as we use Bigquery, even though I have turned on caching on bigquery side, fetching data takes some time and you still pay for egress. Caching in my case is not only speeding up things but reduces the cost as well.

I looked at Redis & Memcache, but I didn't have clarity on how those services will be deployed with our production setup. Also, I wanted to try out the idea first, so I decided to go with "DiskCache" an amazing python module. It uses SQLite to store the content on local file-system. I ran it with 32 processes, and Pandas DataFrames up to 500MB, and It worked without any issues. Data is fetched and put into Diskcace i.e. sqlite and then added a layer of lru_cache on top of that, so during execution same data is served from memory.

Now you have some idea about my caching setup, lets talk about the issue I had and how it came to surface.

In 1.5 years, Framework was mostly used for training 1000s of small models (of similar type, changing target, estimator etc.). Caching was used to store the pandas DFs and some other kinds of objects which were only accessed for reading. As more users started playing around with the tool to try out their ideas and few users complained about randomly getting wrong results. It was not easy to reproduce as on consecutive runs everything worked fine. Till, we found the problem.

As a coding standard in framework, whenever we fetched data (i.e. Pandas DF) and did any kind of processing (transformations) we always saved in a new dataframe. Some users started doing inplace=True out of habit and this not only changed the current result but also the result in the cache. Lets dive deeper into why this happened?

I am going to mimic the issue with dict instead of data frame.

I have a decorator that wraps the function/method with cache. Something like below,

from functools import lru_cache
import time
import typing as t

@lru_cache
def expensive_func(keys: str, vals: t.Any) -> dict:
    time.sleep(3)
    return dict(zip(keys, vals))


def main():
    e1 = expensive_func(('a', 'b', 'c'), (1, 2, 3))
    print(e1)

    # Now, we have e1 in cache, so fetching it again should be instant
    e2 = expensive_func(('a', 'b', 'c'), (1, 2, 3))
    print(e2)

    # Since we store a dict or DF object in the caching what should happen if i change 
    # the fetched value?

    e2['d'] = "amazing"  # should it change anything in cached value?

    # If you answered it no, then you are in for surprise...
    print(e2)

    e3 = expensive_func(('a', 'b', 'c'), (1, 2, 3))
    print(e3)  # you will see that e3 also has a "d" a new key added.


if __name__ == "__main__":
    main()

Try it in playground

Why do see this behaviour? Because lru_cache gives you a reference to cached variable, and when you modify, it modifies the actual value referenced by the pointer.

Solution

Since we know that issues is due to working with reference of a cached value and I have no control over how it will be used downstream. easy (& simple) solution I implemented was to return a copy every time you are returning a result. Since you are working with a copy of data user can change it or do anything with it. This results in data duplication, you might have multiple copies of same data, it was not a problem for us.

I am going to mention couple of points from some architecture books I am currently reading,

No architecture is good or bad, it's how expensive (time-wise) is to make a change.
Architecture is about making decisions with least amount of bad things (based on known-knows, known-unknowns).
It's always a compromise, if you think your solution has no compromises then there is a chance that you have not understood the problem fully or thought about all scenarios (or not yet surfaced - unknown-unknowns).

Now, getting back to solution, all we need is a small decorator which decorates the lru_cache and returns a deepcopy of the cached object every time its accessed

def custom_cache(func):
    cached_func = cache(func)  # evaluated only once!

    @wraps(func)
    def _wrapper(*args, **kwargs):
        return deepcopy(cached_func(*args, **kwargs))  # gets called every time

    return _wrapper

Lessons Learned

Understood lru_cache in bit more detail.
Your recommended approach can prevent bugs.
Users are not going to listen / follow same coding practices like developers do, so try to fix the implementation whenever you can.

Python: Interesting Code Patterns

Ninad Mhatre — Sat, 07 Sep 2024 18:11:01 +0000

I work mostly with Python and review code on almost daily basis. In our code base formatting & linting is handled by CI jobs using black & mypy. So, we focus only on changes.

When working in a team, you already know what kind of code to expect from a certain team member. Code reviews become interesting when a new person joins the team. I say interesting, as everyone has some coding style that they use it unconsciously; for good or bad! Like I have some,

Setting the value of Optionaltype. Usually these variables are part of the signature

# I used (long back) to do

def func(a: int, b: Optional[list] = None, c: Optional[Dict] = None):

    if b is None:
       b = []
    if c is None:
       c = {}

# Instead I do

def func(a: int, b: Optional[list] = None, c: Optional[Dict] = None):

    b = b or []
    c = c or {}

Handling Simple if..elif..else (up to 3-4) statements with dict

This is simple use case where you return a string or call a func based on some value

Note: from 3.10 you should use match instead of this.

Instead of doing,

def get_number_of_wheels(vehicle: str):
    if vehicle == "car":
        wheels = 2
    elif vehicle == "bus":
        wheels = 6
    elif vehicle == "bicycle":
        wheels = 2
    else:
        raise ...

# I prefer doing, 

def get_number_of_wheels(vehicle: str):
    return {
       "car": 2,
       "bus": 6,
       "bicycle": 2
    }[vehicle]       # Raise is now KeyError

Above are couple of examples, people who review my code will have more examples.

Recently, A new developer joined my team and i noticed a pattern which I loved but I asked to change it to simple if..else case. I will show you the pattern first then give my reason for asking for change.

The code is a decorator which does something to parameters. Lets write a simple (useless) decorator which will print number of args & kwargs with which function/method was called.

def counter(is_cls_method: bool = False):
    """ print number of args & kwargs for func/method """
    def outer(f):
        def inner(*args, **kwargs):
            args_cnt = len(args)
            kwargs_cnt = len(kwargs)
            print(f"{f.__name__} called with {args_cnt=} & {kwargs_cnt=}")
            return f(*args, **kwargs)
        return inner
    return outer

@counter()
def test1(a, b, c):
    pass

class A:
    @counter(is_cls_method=True)
    def test1(self, a, b, c):
        pass


print("function")
test1(1, 2, c=3)
test1(a=1, b=2, c=3)

print("method")
a = A()
a.test1(1, 2, 3)
a.test1(1, b=2, c=3)

On running this code, you should see

function
test1 called with args_cnt=2 & kwargs_cnt=1
test1 called with args_cnt=0 & kwargs_cnt=3

method
test1 called with args_cnt=4 & kwargs_cnt=0
test1 called with args_cnt=2 & kwargs_cnt=2

Its working fine but for methods, its also counting self. so lets fix that!

def counter(is_cls_method: bool = False):
    def outer(f):
        def inner(*args, **kwargs):
            args_cnt = len(args)

            if is_cls_method:   # Check added
               args_cnt -= 1    # Reduced count by 1

            kwargs_cnt = len(kwargs)
            print(f"{f.__name__} called with {args_cnt=} & {kwargs_cnt=}")
            return f(*args, **kwargs)
        return inner
    return outer

This is a simple if clause, but new developer did something else which was interesting use of boolean.

I am showing only the changed code...

   args_cnt = len(args[is_cls_method:])

Solution is lot better than using if, as bool in python is just int. Original code was bit longer and noticing this small change is not obvious, also code base used by users who are basic python users. And, if you have to guess what a line is doing, i think you should change to make it obvious.

What is your thought on this, Do you use boolean as index?
Do you have any more python patterns like this?

Python Environment Setup

Ninad Mhatre — Mon, 12 Aug 2024 19:33:25 +0000

I am a python developer and usually work with multiple repositories at any time. Most of these projects have different dependencies so I create a virtual environment in every project. I prefer creating virtual env right next to code. Pycharm also detects it when repository is opened in Pycharm.

This setup worked most of the time but sometimes, I activate venv of one repository and then change it another repository without changing activating correct environment. I usually spotted this when code execution used to fail and never lost more than a minute on this but, this always made me think what if there is a shell that could run something when you enter a directory?

I decided to implement something simple in shell script temporarily. I ended up with creating small function, aliased with cd and added in my .bashrc.

It's a simple function and all I ever needed. It does,

Loads the .venv from that repository
Additionally, sources .cdenv, i usually set env variable but you can do anything in there.

I was okay with not deactivating/resetting environment when i exited the directory.

Add below function in .bashrc and you should be able to use it.

Note: If you create your virtual environment with different name then please change the name in code below.

function cd() {
  # Run a shell `cd` command or error
  command cd "$@" || return

  # If .venv exists in the directory
  if [[ -d .venv ]]; then
    # Deactivate any initialized virtual env, ignore error
    deactivate > /dev/null 2>&1

    # activate .venv
    source .venv/bin/activate
    echo "deactivate prev venv and activate current venv"
  fi

  # Load a .cdenv file
  if [[ -f .cdenv ]]; then
      echo "loading .cdenv"
      source ./.cdenv
  fi
}

Estimating Coding Tasks: What Could Go Wrong?

Ninad Mhatre — Sat, 10 Aug 2024 10:27:47 +0000

Here is how the task of “adding a hash to an existing DataFrame” went from taking a couple of days to consuming almost an entire sprint.

In Q2 of 2022, I started working on a data pipeline that fetches market data from a REST service and stores it in a BigQuery table. This is a high-level explanation of the pipeline. The interesting part is how the data is queried data, converting it to DataFrame and then uploading it BigQuery tables using GCSToBigQueryOperator of AirFlow.

Initially, it seemed simple to write, but Airflow’s “idempotent” principle added it bit of challenge. What to fetch from this REST service was decided by another table and even if JOB is idempotent the table it used as reference could change between 2 runs. After spending additional time, talking to Data Engineers pipelines was ready By the end of Q3 of 2022.

Fast forward to Q1 of 2024. By this time, we had more users accessing the data, and we realized our query pattern was not using partitions properly. Or, rather, we wanted to access the data based on a string column, but it’s not possible to partition on a string column in BigQuery. This led to scanning large amounts of data and frequently hitting the daily quota.

This led us to consider how to partition data based on string columns. Our data engineer suggested converting that string column into an integer using FarmHash with an additional modulo operation. In the proof of concept, this reduced scanning by almost 90%, and query performance increased by 3-5x. We decided to proceed with this as the final solution. All we needed was to:

Create a table with Farmhash fingerprint
Change pipeline to compute the fingerprint
Upload the data.

To compute FarmHash fingerprints in Python, there is a pyfarmhash module. I installed the module and used the code below to compute the hash, and locally it all worked as desired.

def get_hash(val: str) -> int:
    return additonal_logic(pyfarmhash.fingerprint64(...))

df[‘hash’] = df[‘Col’].apply(get_hash)

With all the tests passing, it was now time to push the code to Airflow and run it. I was not expecting anything to go wrong at this stage. In fact, I was happy that everything worked as planned and within the estimated time.

With a happy mind and full of confidence, I pushed my changes, started the job, and then waited for 10-15 minutes for it to finish. Meanwhile, I switched to another task. Soon, I received an unexpected failure email from Airflow. I looked at the logs and was surprised to see that it failed while installing the pyfarmhash module!

To help you understand the problem, I need to explain the structure of the job. The job has the following steps:

Download the data in parquet format
Upload to GCS bucket
Delete existing data; if any. (avoid duplicate data)
Upload the data to BQ tables.

In this process, task-1, which downloads the data, is a separate Python module. To run it, I used the PythonVirtualenvOperator from Airflow. This operator allows you to specify packages as requirements and then installs them in a newly created virtual environment. Once the package is installed, all its dependencies are also installed, and you are ready to roll.

I added pyfarmhash as a dependency to the module that downloads the data, and everything else remained unchanged. And it failed! Why?

pyfarmhash is a hashing library implemented in C/C++. Upon installation, it requires GCC to compile the package, and that was not present on the Airflow host. It made sense not to have GCC on the Airflow host, but unfortunately, this was a blocker for me.

I looked for a pure Python implementation of the pyfarmhash package, but there were none. Then, I looked for wheel packages, but again, there were none. I considered building wheel packages and pushing them, but that would have led to a long-term responsibility of providing wheel packages internally. I wanted to avoid additional, workaround-like steps. I explored all the options and discussed them with the team maintaining Airflow. They suggested creating a Docker image and running it in KubernetesPodOperator. This was a good option, as I could control the environment and include whatever was required without relying on an external environment. Additionally, this solution had no workarounds. The only short-term drawback was that it needed more time to implement.

Before starting with a Docker-based solution, I had already spent about 16-20 hours on this task. For the Docker-based solution, I additionally needed to:

Change the Python package to have entry points to start download and purging logic.
Create a Docker package and test it (this was my second Docker image).

Since I was no longer going to use PythonVirtualEnvOperator in Airflow, I decided to remove it completely also improve the workflow. I had to change the python package to have entry points to start download & purging logic

It took me an additional 30-36 hours to have a final solution with the Docker image ready, which is 6-7 working days and, with initial 2 days included, it became a sprint long task.

I look back on this and wonder, I had to throw away working solution, change the module structure, create a docker image, change 10+ AirFlow jobs to use Docker image for tasks, dealing with this reality and overcoming initial frustration. All of this only because, "A single python module required “gcc” to compile!"

Towards cloud native dev...

Ninad Mhatre — Sun, 25 Sep 2022 11:22:04 +0000

I am a Python developer with 15+ years of experience. I was working as automation QA engineer (created custom testing framework, modifying it and doing functionality testing) for 4+ years and as you would have guessed it, working on something for 4 years surely does take out charm / challenge out of it. So i was looking for new things and at the start of 2022 there was email about Python developer role in some other team. I jumped onto it and got it just based on recommendation.

So, this new role was with Quantitative research team, just to give my background, I am not good at math, specially math which only has letters in their equation :). One thing i was sure about my skills and i knew that i would learn anything for the task i am working on and produce result and improve them as i get more experience / knowledge.

I was happy that i will work with Scientific side of Python libraries (numpy, scipy etc.) and then with GCP (Google Cloud Platform). First task i started with was creating Pipeline to load data into bigquery for our ML Pipeline. I faced some design decision issues just because i was not following cloud native approach.

Get requests to data server timing out after first 1000 requests.

I created fully functional local Airflow pipeline but it started timing out on GCP. Why? cause i was opening new socket for every request to save host! This would have worked fine only any VM or server but not on cloud. Though, i think i should have used Session from the start. Learned why to use sessions

Re-running airflow job

I must confess that choosing BigQuery (BQ) was not my decision. As BQ has no concept of duplicate / unique row concept, we needed to make sure that if we re-run job for some day it needs to first clean the data from BQ. So I was forced to think in a different way of how to structure the data, partition on column and then in case of re-run just drop the partition. But how to decide which partition to drop? all these decisions impacted my conventional developer thinking and helped me to think like Cloud-Native developer.

Do the backfill with starting date going back to 6 years back.

This i must confess was the hardest part of the whole pipeline. How would you one-off back-fill for past 6 years and then do it periodically if some data is corrupt for specific IDs.

When i started with this development, I was very sure that this whole thing can be completed in 1 month but in reality, I had to learn about Airflow, Bigquery, GCP limitations and access rights! In the end, i spent close to 2.5 months to get this pipeline running in production. In the end, i was very happy that my first pipeline is working fine for 3+ months without having any issues.

Good enough is fine...

Ninad Mhatre — Sun, 03 Jan 2021 14:35:46 +0000

Still remember vividly, myself coding for some resume creation site in hospital's waiting room while my wife was doing her prenatal exercise. I was working on this site for 6 months by then but it was no where close to being ready for release in the wild! And unfortunately i was responsible for keeping it that way, like working on Jira like site so that i can create new bugs/improvement tickets easily and wasting one week behind it or pushing to create better UI and kept on delaying the release till Feb 2017, when my son was born and i don't remember working on this site for next 2 years. In 2 years, I not only lost the momentum but also the interest to continue! Its been 4.5 years since i first started with this idea and this site is lying somewhere on my private repository. I no longer even wish to publish it!

Fast forward to Nov 2020, saw my wife who is very active on Instagram managing her hashtags in keep by creating multiple notes and then copy pasting it. There i got the idea of save tags. A website where you can categorize tags and then select them based on your post. Main reason for creating was just to create something but i still wanted it to be prefect but i learnt from last experience, so this time i just created site, wrote some tests for back-end, decent UI (i am back-end dev so i just used Bootstap and google), use only google oauth (last time i setup google, facebook, twitter and github) and good to go. I sent it to couple of friends but no one gave any feedback ☹️ something to keep in mind for next time. This site is created just for fun and not with intention on making business out of it so give it a try and share your feedback.

Perfect product do not exist! Close to perfection takes time and good enough is okay for the most and you get valuable feedback which helps to shape product. In case you have half baked product try to release it as soon as possible.

Updated:
~~Lastly, please checkout https://save-tags.com~~

I have taken down the site after running it for 3-4 months. But point of the post was, If you create something release it once you think it's good enough! You might never achieve the perfection you are looking for.

software testing, assertions

Ninad Mhatre — Sat, 11 Jul 2020 15:16:43 +0000

I work on a testing framework which validates application by sending and receiving messages with the proprietary protocol. And certainly, there are a lot of assertions with most of them checking the value of the field, some checks the type of message received and some ordering of messages. All of these assertions are grouped by functionality in multiple functions across multiple files. Having a lot of assertions is not a problem but not showing a clear message is certainly a problem. Imaging test being failed with something like this

... long stack trace ...
AssetionError: True != False

looking at it just gives a basic idea that something should have been True but was not! If you read stack trace you can figure out where the assertion failed, but there is a big "if you read" Nobody reads it! Then we get a lot of links to failed jobs with a sweet message, "tests are failing".

This is due to having lot of bare asserts, when we started developing the tests our the main focus was converting manual tests to automated and that's why we have lot of asserts lying around in our code base.

The second pain point was not that big deal, but surely good to address is, when we check new message structure and values, its fails one by one. If message has 20 fileds then you can imagine how long it will take to figure out all wrongly populated fields. I wanted to report all failures at once so that I can ask a developer to take a look at all in one go. There is already a solution to this, delayed_assert module in python, but that is a wrapper around assert which stores stack trace and dumps in the end and it also meant that i need to replace all assets.

I decided to solve both the problems in one go, I searched and found few assertion libraries in python, one particular I liked was assertpy because it has clean API and it has assert_warn function which would just report the errors (without stack trace) but won't stop the execution. My idea was, write a wrapper which would return assert_that or assert_warn depending upon a certain condition.

# some wrapper file
import os
from assertpy import assert_that, assert_warn

def get_assertor():
   if bool(os.getenv("ERRORS_AS_WARN", False)):
       return assert_warn
   else:
       return assert_that

assert_that = get_assertor()

also the API os assertpy is just brilliant

from assertpy import assert_that
from nose.tools import assert_equal

# not very easy to understand
assert_equal(1 + 3, 2, "4 is not equal to 2")

# its like reading a sentance
assert_that(1 + 3, "addition").is_equal_to(2)

I decided I will use assertpy but then I started implementing and I realized,

1) we only use a handful of assertions and regular ones that we use are not in assertpy

2) though assertion methods can be easily plugged in assertpy, requirement of such function is the first argument named as self, which felt bit odd

def is_in_between(self, other): 
   ...

so I checked the code and realized that for every assert_that, a new instance of the class is created and extensions are added on every initialization, I thought its extra work for little gain.

3) my dynamic error to the warning was working but it was printing the wrong location of the source as it was looking for assert_warn in stack trace but I was referring it to as assert_that, in short it was only working partially.

Adding PR to assertpy seemed worthless as all solutions felt like a hack and then I decided to write one which is based on assertpy but has very basic functionality but easy to extend.

I created simple-assertions, after thinking many times whether to re-invent the wheel or not...

how I solved the above issues?

1) I created a class with a handful of common assertions, the class was picked as you can easily inherit to extend the functionality.

from simple_assertions import SimpleAssertions
from typing import Union

class YourAssertions(SimpleAssertions):
    def __init__(self, as_warn=False, logger=None):
        super().__init__(as_warn, logger)

    def is_greater_than(self, other: Union[int, float]):
        if self.val_to_chk.val < other:
            self.error(self.generate_err_msg(other, "to be greater than"))
        return self


class YourTestClass:
    def __init__(self):
        self.assert_that = YourAssertions().assert_that

    def test_something(self):
        self.assert_that(4 + 10).is_greater_than(10).is_equal_to(14)
        self.assert_that(1).is_instance_of(int)
        self.assert_that(3, "lucky_num", as_warn=True).is_equal_to(4)

2) With composition, an instance of assertion class can be created and then that single instance can be reused. IDE can also help me code completion.

class MyTests(unittest.Testcase):
    def setUp(self):
        self.assert_that = SimpleAssert().assert_that

3) I changed the API of assert_that and instead of 2 functions, 1 for error and 1 for the warning, I added an optional parameter to create warnings, also added an environment variable which if set will convert all error to warnings.

Use of environment variable to change the behaviour is topic for discussion but for my needs, it was a perfect solution!


# instead of, assert_warn(10).is_equal_to(9)

assert_that(10, as_warn=True).is_equal_to(9)

I am planning to add a few more assertions so that it can be used without need to adding anything. If you want to give it a try, all you have to do is

pip install simple-assertions

and you can check the source here please check here https://github.com/ninadmhatre/simple-assertions

rusty start...

Ninad Mhatre — Sun, 21 Jun 2020 21:56:38 +0000

I heard about rust somewhere in 2017 but that time i picked Go over rust and started reading about it. Go syntax was simple compared to rust and Go seemed natural choice for someone who is coming from Python.

I liked Go but never loved it, that was the reason i could not keep up the momentum to learn it for a long time. Eventually i gave up and just focused on current job and getting used to new country (i moved to new country for job). Once things settled and decided to look for new challenges there were 2 options, Data Science and Some compiled language!

As any sane person with Python Exp. would do, i picked DS and started reading about it but this decision was more from job point of view and not something i wanted to learn so then i was back to square one, which language to pick? Go or Rust or something else? I decided to give rust another chance somewhere towards the start of 2019 and seemed like i read "rust" for the first time, i didn't find it difficult (to read), compiler was nice and official site has nice website explaining entire language. so as you guessed i picked "rust". After reading about some informational material i decided to jump directly into coding.

You never learn driving by reading a book or for that matter you learn nothing just by reading about it.

I decided to convert one small shell script which is a wrapper to start/stop/check 2 binaries. Shellscript was perfect for this but this was better task than solving some coding challenge questions. i learned a lot about the language, specially compiler is amazing, errors are clear also it says how you could correct them! I wrote something which worked fine and got it reviewed by putting it on rust community and guess what, ppl actually reviewed the code and gave me inputs. There i actually fell in love with this entire setup then fast forward to 1 year to Feb 2020. In this last 1 year i could never reach beyong chapter 9 of "rust by example" something or other used to come in between and i had to refresh everything due to gap of month or so.

2020 was the year i was looking forward to as i had made some plans but Corona had another plans and my plan was pretty selfish so it didn't work out! that gave me opportunity to go back to learning things, now i am on chapter 14! :). Slowly & steadily i am moving but there is a progress which is important.

While coding it's important to have a nice setup where things just works and that was my problem, due to Corona, i was using my laptop to connect to office and since Webex has no Linux client and to avoid many setup issues i decided to boot into windows 10 since March and it will stay like that till i don't join office back. This was a issue, i dont really like windows for developing anything (C# was only exception!). I spin my virtual machine and code in that or use WSL but for rust i was not able to find the setup i could work with. Swithing to linux disto was not an option (my grub does not work and i have to change BIOS everytime) so i downloaded Lubuntu but it was painfully slow with 2.5GB ram as a last resort i decided to pick WSL and VS Code which connects to WSL. I have autocomplete for now and i compile manually with Cargo but it's lot better than other setup i tried.

Once i am done with all the chapters maybe in a month, i will write some code around client-server architecture and improve on that iteratively to learn the language, but till then i have to go on with steady pace...

this should be a separate PR!!

Ninad Mhatre — Tue, 07 Apr 2020 21:39:18 +0000

Recently while working on some task I had to write some hacky code, and by hacky, I mean code with lot if..else. There are a lot of places in the code base with such random if..else changing the flow of based on some environment variable. Hence I was initially okay with adding all these ifs but once I finished with the functionality and looked back to refactor so that I can send it for review, no matter how I refactored it, it turned out to be messy code which I just couldn't push for a review!

Then I took a decision to refactor whole functionality so that I can later remove all the ifs placed at multiple places in code. When I decided to refactor it was good from a long term perspective just that I decided to take such a decision without discussing it with anyone in the team. Before you get me wrong, this refactoring was already done to new parts of the code to solve this exact problem just that it was not applied to the old part of the codebase due to not having enough time or not the correct opportunity.

Once I pushed it for review, the first comment I got was "this should be put in separate PR!". Well, in all honesty, that was expected and also true! but in reality, due to 1 QA per 5 devs, my team is usually bottleneck for releases and hence we hardly get any time to work on old code or a new feature in our test framework. If I had left the refactoring for later stage I would have never got another opportunity to work on that part of the code but the part I really wanted to avoid was leaving the code with if..else which is really hard to understand after some time like for e.g. why was this different? why I am converting variable to int and sometimes to float? or some similar questions. Of course, I can add comments but how many such comments one can add?

From a regular development environment, I should have raised separate PR but while working in a QA, with stringent timelines I think going ahead with single PR with "unplanned" changes was the right call!

Just snippet of what code looked like (all are made up fields), and what I refactored it into...

Old Way

# checking request against response or some constants

assert_equal(request.name, response.name)
assert_equal(request.type, response.type)
assert_equal(response.checksum, get_checksum(request))

if env == "DEV":
    assert_true(respone.timestamp)
else:
    assert_ts_in_range(response.timestamp, tolerance_in_sec=2)

New way

assertor = get_msg_assertor(response, env)  # class based

assertor.name(request.name)
assertor.type(request.type)
assertor.checksum(get_checksum(request))
assertor.timestamp()  # DEV will just check for presence and UAT will check for tolerance