DEV Community: Sarath

Effective Bash for effortless insights

Sarath — Sat, 29 Mar 2025 08:22:42 +0000

Problem: In my company, I was given the following problem.
Get the users who are inactive i.e whose user_activity data isn't present and they have paid. This is a simple problem, but the way I solved using some clever tricks.

Context: The user activity data first gets into an s3 bucket, later it gets processed and puts into a mongo collection user_activity.

Stages of solving the problem:

Get all the unique users from s3 using athena.
Get all the users from the mongo collection.
Get the difference between them.

Getting the data from s3

create table user_data from "s3:user_data";

select distinct(user_id) from user_data;

This I downloaded as csv and saved it in the following path: /tmp/user_ids_in_s3.csv

Getting the data from mongo

This was the tricky part because of the way the data is stored and the exports of mongo.
We have the following tables:

user_activity stores all the user activity.
user_details stores the details of the users if he paid or not.

Pipelines to the rescue. I used mongodb pipelines to get the data of paid users user activity.
The pipelines are something like this

[
  {
    $lookup:
      {
        from: "user_details",
        localField: "user_id",
        foreignField: "_id",
        as: "result"
      }
  },
  {
    $match:
      {
        "result.status": "paid"
      }
  },
  {
    $project: 
      {
        _id: _id
      }
  }
]

exported this data as json into /tmp/db.user_activity.json

Where's bash ?

Good question! The way I got the data was in the following format.

[{
  "_id": {
    "$oid": "667..."
  },
 }
 {
  "_id": {
    "$oid": "867..."
  },
 }
...
}]

Now, this we are probably going to compare with csv.
Therefore it'll be better if the file was a little nice with just user id's.

Bash's rescue

Getting the user ids from the data in processable format

cat /tmp/db.user_activity.json | 
jq '.[]._id."$oid"' | 
sed 's/"//g' > paid_users.txt

Explination:

cat /tmp/db.user_activity.json outs the file to stdin.
jq '.[]._id."$oid"' print's the oid field from the json.
sed 's/"//g' cleaning the quotation marks in the data.
> paid_users.txt output the whole thing into a file.

If you don't understand any part feel free to read the docs of each command:

I highly encorage you to work out this command using the following data.
Use this script to generate data:

✨ This script is generated by AI.

import json
import random
import time

def generate_object_id():
    timestamp = int(time.time()) 
    random_bytes = random.getrandbits(8 * 8).to_bytes(8, 'big').hex()  
    return f"{timestamp:08x}{random_bytes}"[:24]

def generate_data(count=5):
    return [{"_id": {"$oid": generate_object_id()}} for _ in range(count)]

if __name__ == "__main__":
    data = generate_data(10)
    print(json.dumps(data, indent=2))

Comparing "paid users" with "active users"

comm -23 <(sort /tmp/user_ids_in_s3.csv) <(sort /tmp/paid_users.txt) | 
sed -E 's/[[:space:]]+//g' | 
sort | uniq

Explination:

comm -23 <(sort /tmp/user_ids_in_s3.csv) <(sort /tmp/paid_users.txt) find the files which are present in one but not both.
sed -E 's/[[:space:]]+//g' formatting the data as it has spaces at start and end
sort | uniq sorting the data and only taking the uniq id's.

If you don't understand any part feel free to read the docs of each command:

Worthy mention for getting the user ids from the mongo data

My collegue had to do something similar. What he did completely blew my mind.
I never tought of this way so I'm sharing his approach as well.

He opened the file in vscode and used a regex "$oid": "(.*)" selected everything using ctrl+f2 and copied them to clipboard.
Pasted this in a new file used alt+shift+end which goes from the cursors location to the end of the file,
deleted quotations along with the pesky $oid.

It blew my mind I had never tought of it, altough inefficient and prone to errors. Hat's off.

How to Hide Local Git Changes Without Ignoring Files

Sarath — Sun, 16 Mar 2025 08:55:00 +0000

TLDR;

git config --global alias.hide 'update-index --assume-unchanged'
git config --global alias.unhide 'update-index --no-assume-unchanged'
git config --global alias.hidden '! git ls-files -v | grep '^h' | cut -c3-

Motivation

Has this ever been you problem. You frantically type some code and you have changes in 10 of files and you only need to commit just one of them and all the other files are configuration files or something else that's unique to your machine and this doesn't need to go to the source control.

Well, recently I was going through the same problem. I was using a windows machine and my co-workers with a mac. All of them use mac therefore the npm-scripts, linting configuration, dockerfiles etc... are only mac specific. Therefore, I have to change them to work for my machine. This doesn't need to be pushed to source control. And they are inheritly against choosing an agnostic solution. Therefore, I have almost 10-12 config files which are specific to my machine. But this messes up with my git status. I wondered is there a way out of this mess. Then Searching through the Internet I found update-index which is a command in git, it helped me with my problem therefore, I am sharing this with you guys.

Solution

There is a concept of index in git. Basically you can think of index as a snapshot of the repository which git tracks. Basically any command like status, add ..etc will use this index to perform operations with git. So, we just need to update this index. And git generously provides us with this command called update-index. Which we can use to udpate the index of the git. This inturn will make the changes doesn't appear in git status or anything similar. Which is exactly what we need.

git update-index --assume-unchanged file.txt

Ok, this is all great but what if, I want the changes in the file to be commited how to do that ? As we know that we have updated the index to reflect that the file has no changes we can make sure to say that the file has changes. That can be done in the following way:

git update-index --no-assume-unchanged file.txt

This will ensure that this file is tracked by git.

But what if you want to see the hidden files ?

git ls-files -v | grep '^h' | cut -c3-

Add the commands as aliases

git config --global alias.hide 'update-index --assume-unchanged'
git config --global alias.unhide 'update-index --no-assume-unchanged'
git config --global alias.hidden '! git ls-files -v | grep '^h' | cut -c3-

Usage

Rather than typing the long command git update-index --assume-unchanged file.txt as we have already added this as alias called hide we can do something like this:

git hide file.txt # same as git update-index --assume-unchanged file.txt
git unhide file.txt  # same as  git update-index --no-assume-unchanged file.txt
git hidden # same as git ls-files -v | grep '^h' | cut -c3-

Refrences

update-index docs.
Inspired from github